Intrinsic Evaluation of Grammatical Information within Word Embeddings
PACLIC 33
Daniel Edmiston, Taeuk Kim
One-Line Summary
A proof-of-concept framework for intrinsically evaluating grammatical (rather than lexical) information in word embeddings: six Korean embedding models are distilled into three morphologically-aware learner networks, and the resulting affix representations are compared against a hand-built, 62-dimensional discrete feature space over 107 Korean affixes. All models beat a random baseline, syntactic information is captured even by linear affix representations, semantic information requires non-linearity (up to 33% overlap vs. 2% random), and no model is sensitive to phonological or pragmatic features.
Figure 1. The StaffNet learner architecture showing the dynamic composition of cheese-burger.PL.NOM. Stems ("cheese", "burger") are fed through a BiLSTM and combined by a convex combination; affixes (PL, NOM) are then applied iteratively as matrix transformations on the compound stem representation.
Background & Motivation
Neural NLP systems have dominated the field, but their improved performance has often come at the cost of understanding what the models actually learn. Work on natural language inference (McCoy et al., 2019) showed that carefully chosen examples can fool such systems, suggesting they learn dataset heuristics rather than genuine linguistic generalizations. Methods for probing the information encoded in these models are therefore of growing importance.
Existing intrinsic evaluation methods for word embeddings (e.g., the word analogy task: king is to queen as man is to what?) overwhelmingly test for lexical semantics. Extrinsic probing methods, which train classifiers on top of fixed representations, have examined morphological and syntactic content, but they measure linguistic knowledge only indirectly, through the performance of a downstream classifier.
Key Idea: Instead of asking whether embeddings know that Paris relates to France, ask whether their geometry reflects grammatical properties. The paper compares continuous embeddings with ground-truth embeddings living in a linguistically-inspired, discrete binary feature space -- the corners of an n-dimensional hypercube -- so that similarity between the two spaces can be read as grammatical, rather than lexical, content.
Why Korean affixes? (i) Korean is agglutinative, so there is largely a one-to-one mapping between affixes and meanings; (ii) its morphology is highly regular, enabling high-fidelity parsing; and (iii) Korean affixes encode at least four distinct types of information: phonological (allomorphy such as accusative -을 after a coda vs. -를 after a vowel), syntactic (place of attachment), semantic (focus markers, logical operators, case), and pragmatic (formal register, honorifics). This lets the authors run focused tests on which information types models are sensitive to.
A further obstacle is that most embedding models (Word2Vec, GloVe, even character- and syllable-level models) have no explicit notion of an affix, so there is no affix vector to inspect. The paper resolves this with transparent model distillation (Tan et al., 2017, 2018): it trains a morphologically-aware learner network to reproduce the black-box embeddings, and then inspects the learner's affix parameters as a proxy for what the original model captured. To the authors' knowledge, this is the first application of transparent model distillation for interpretation in an NLP setting.
Proposed Method
The pipeline has four stages: train ground-truth embedding models, distill each into morphologically-aware learner networks, build a discrete ground-truth feature space for affixes, and compare neighbourhoods across the continuous and discrete spaces.
1
Ground-Truth Embedding Models
Six Korean embedding models spanning three granularities are trained on a Korean WikiDump with vocabulary limited to 10,000 words and 300-dimensional embeddings (following Choi et al., 2016). Character-level: fastText (Bojanowski et al., 2017) and Naver's kor2vec (based on Kim et al., 2016). Syllable-level: the model of Choi et al. (2017). Word-level: Word2Vec Skip-gram, Word2Vec CBOW (Mikolov et al., 2013), and GloVe (Pennington et al., 2014). Other hyperparameters follow the original publications.
2
Transparent Model Distillation into Three Learner Networks
Each ground-truth model is distilled into three learner architectures that explicitly compose words from morphemes (parsed with the Komoran POS-tagger from KoNLPy). The learner iterates over the same corpus, computes its embedding ŷ for each word, and minimizes the squared distance to the ground-truth embedding y. StaffNet (Edmiston and Stratos, 2018) treats stems as vectors (combined via a BiLSTM and convex combination) and affixes as linear functions, i.e., 300×300 matrices applied iteratively to the stem. MRNN (Luong et al., 2013) is a recursive network over the binary morphological parse, computing parents as p = tanh(W[c1; c2] + b). Tree-LSTM (Tai et al., 2015) is an N-ary Tree-LSTM with N = 2 over the same binary parses. In MRNN and Tree-LSTM, affixes become 300-dimensional vectors.
3
Discrete Ground-Truth Feature Space
A hand-constructed dataset embeds 107 Korean affixes into a binary feature space {0, 1}62, where each dimension is a linguistic feature such as [+CODA] or [+HONORIFIC]. The 62 features are partitioned into four subsets: phonological ([±CODA], [±LOW]), syntactic ([+POSTPOSITION], [+CONJUNCTIVE], [+X-LIMITER], [+Z-LIMITER], [+V1]..[+V4]), semantic (46 features such as [+NOMINATIVE], [+TOPIC], [+PAST], [+CONDITIONAL], [+QUOTATIVE], [+ONLY]), and pragmatic ([±FORMAL], [+HONORIFIC], [+FAMILIAR]). Across the affixes there are 5 distinct phonological, 8 syntactic, 55 semantic and 5 pragmatic configurations. Distance in this space is Hamming distance dH.
4
Similarity in the Continuous Space
For vector-valued affixes (MRNN, Tree-LSTM), cosine similarity is used. For StaffNet's matrix-valued affixes, cosine similarity does not apply, so the paper uses subspace similarity (Mu et al., 2017): take the first N principal directions of each affix matrix, stack them into X, Y ∈ Rd×N, and score the pair by the root mean of the squared singular values of XTY, yielding a value in [0, 1]. The intuition is that similar affixes should map stems to similar subspaces. N is chosen as the minimum number of principal components explaining 25%, 50% or 75% of the average affix variance for each model.
5
Nearest-Affix Evaluation with Two Graded Scores
For each affix, the set YAff of its closest affixes under Hamming distance in the discrete space is computed (size k), and the k most similar affixes in the continuous space form ŶAff. Two scores are reported: percentage of overlap between the two sets (equivalent to an F1 measure on cluster analysis since the sets have equal cardinality), and the Hamming offset -- the average Hamming distance from the affix to the continuous-space neighbours minus the minimum possible distance. Unlike all-or-nothing analogy tests, both measures are graded. The same procedure is re-run restricting the discrete space to each feature subset (phonological, syntactic, semantic, pragmatic), and compared to a random baseline.
Experimental Results
Verifying the Distillations
Before evaluating, the paper checks that the learners faithfully reproduce the original embeddings. Average cosine similarity between ground-truth and distilled embeddings over the 10k vocabulary is high for all combinations, which is notable given that almost all of the volume of R300 is orthogonal to any given point.
Model
StaffNet
MRNN
Tree-LSTM
kor2vec
0.971
0.840
0.952
fastText
0.949
0.824
0.946
Syllable
0.967
0.869
0.979
W2V-SG
0.953
0.755
0.920
W2V-CBOW
0.941
0.786
0.878
GloVe
0.935
0.690
0.875
Closest-Affix Task over All 62 Features
Percentage of overlap (higher is better) and Hamming offset (lower is better), averaged over all 107 affixes. StaffNet is shown for the subspace rank explaining 25% of variance (its best setting); the 50% and 75% settings score slightly lower.
Model
StaffNet-25% Overlap
StaffNet-25% Offset
MRNN Overlap
MRNN Offset
Tree-LSTM Overlap
Tree-LSTM Offset
kor2vec
12%
3.35
16%
2.82
19%
2.62
fastText
13%
3.23
18%
2.59
16%
2.63
Syllable
13%
3.18
17%
2.88
17%
3.29
W2V-SG
12%
3.17
18%
2.59
15%
3.01
W2V-CBOW
13%
3.19
18%
2.63
16%
2.86
GloVe
10%
3.30
16%
2.80
11%
3.13
Random
2%
4.23
2%
4.32
1%
4.35
Semantic Feature Subset
The sharpest contrast between learner architectures appears when the discrete space is restricted to the semantic features (55 distinct configurations). Linear StaffNet affixes stay at the random baseline, whereas non-linear MRNN and Tree-LSTM affixes reach up to 33% overlap.
Model
StaffNet-25% Overlap
StaffNet-25% Offset
MRNN Overlap
MRNN Offset
Tree-LSTM Overlap
Tree-LSTM Offset
kor2vec
6%
2.54
23%
1.85
25%
1.80
fastText
6%
2.39
33%
1.65
24%
1.72
Syllable
5%
2.47
18%
1.96
19%
2.04
W2V-SG
5%
2.48
26%
1.77
29%
1.78
W2V-CBOW
6%
2.56
24%
1.89
19%
2.04
GloVe
5%
2.49
24%
1.89
23%
1.92
Random
3%
2.79
2%
2.72
2%
2.67
Summary across Feature Subsets
Percentage-of-overlap ranges for the six embedding models versus the random baseline, by feature subset. Because the average neighbourhood size k differs greatly across subsets (e.g., avg. k = 2.21 for all features but 73.91 for pragmatic), performance must be read relative to the random baseline rather than in absolute terms.
Feature Subset
Avg. k
StaffNet Overlap (models)
MRNN / Tree-LSTM Overlap (models)
Random Overlap
Phonological
29.93
29-32%
28-32%
26-29%
Syntactic
30.92
35-41%
34-39%
28-30%
Semantic
3.25
4-6%
18-33%
2-3%
Pragmatic
73.91
68-71%
69-72%
69-71%
Distillation is faithful: Every ground-truth model is reproduced with high average cosine similarity (0.935-0.979 for StaffNet and Tree-LSTM; 0.690-0.869 for MRNN), so the distilled affix representations can stand in for the original models
All models beat random on the full task: Overlap of 10-19% versus 1-3% for the random baseline, and Hamming offsets of 2.59-3.57 versus 4.23-4.47, show that the geometry of discrete grammatical features is partly reflected in continuous embeddings -- though no model reaches high absolute accuracy
Non-linear learners extract more: MRNN and Tree-LSTM outperform StaffNet by a large margin on the full task; the authors attribute this to StaffNet's affixes being purely linear maps applied post-non-linearity
Syntax is linearly capturable: All architectures exceed the random baseline on syntactic features at similar levels, with a StaffNet distillation (fastText, 75% variance) achieving the highest score of 41%, suggesting non-linearity is not required for syntactic relations
Semantics requires non-linearity: StaffNet distillations sit at the random baseline (4-6% vs. 2-3%), while MRNN and Tree-LSTM reach 18-33% (best: fastText distilled into MRNN at 33%), indicating that grammatical meaning of a semantic nature is captured only through non-linear composition
No sensitivity to phonology or pragmatics: All models perform virtually the same as random on both subsets; surprisingly, character- and syllable-level models do no better than word-atomic models on phonological features, and the Wikipedia training corpus is nearly devoid of pragmatically-marked language such as honorifics
Why It Matters
This paper is an early, linguistically-grounded attempt to move intrinsic evaluation of word embeddings beyond lexical semantics toward grammatical information. Its contributions and implications include:
A new kind of intrinsic evaluation: By building a discrete, interpretable feature space and comparing neighbourhoods across spaces, the method yields graded, feature-specific diagnostics (percentage overlap and Hamming offset) instead of all-or-nothing analogy accuracy. The feature-subset design makes it possible to ask precisely which kind of linguistic information a model encodes.
Transparent model distillation for NLP interpretability: The paper shows that distilling a black-box embedding model into a morphologically-aware learner is a viable way to induce representations for linguistic units (affixes) that the original model never explicitly represented -- the first such use of transparent model distillation in NLP, according to the authors.
Different information types, different geometry: The finding that syntactic information is captured by linear relations while semantic information needs non-linearities echoes theoretical linguistics, where semantic analyses typically require more complex algebraic structures built on top of syntactic parses.
Attention to an under-studied, typologically distant language: Korean's agglutinative, highly regular morphology and multi-faceted affix system make it an ideal testbed, and the 107-affix, 62-feature resource constructed here is itself a contribution.
Honest scoping: The authors present this as a proof-of-concept whose principal takeaway is that the evaluation is possible and reveals interesting characteristics of neural representations; they also note that pragmatic insensitivity may partly reflect the academic register of the Wikipedia training data.