EN KO
← All Publications

Intrinsic Evaluation of Grammatical Information within Word Embeddings

PACLIC 33
Daniel Edmiston, Taeuk Kim

One-Line Summary

A proof-of-concept framework for intrinsically evaluating grammatical (rather than lexical) information in word embeddings: six Korean embedding models are distilled into three morphologically-aware learner networks, and the resulting affix representations are compared against a hand-built, 62-dimensional discrete feature space over 107 Korean affixes. All models beat a random baseline, syntactic information is captured even by linear affix representations, semantic information requires non-linearity (up to 33% overlap vs. 2% random), and no model is sensitive to phonological or pragmatic features.

StaffNet architecture composing cheese-burger.PL.NOM
Figure 1. The StaffNet learner architecture showing the dynamic composition of cheese-burger.PL.NOM. Stems ("cheese", "burger") are fed through a BiLSTM and combined by a convex combination; affixes (PL, NOM) are then applied iteratively as matrix transformations on the compound stem representation.

Background & Motivation

Neural NLP systems have dominated the field, but their improved performance has often come at the cost of understanding what the models actually learn. Work on natural language inference (McCoy et al., 2019) showed that carefully chosen examples can fool such systems, suggesting they learn dataset heuristics rather than genuine linguistic generalizations. Methods for probing the information encoded in these models are therefore of growing importance.

Existing intrinsic evaluation methods for word embeddings (e.g., the word analogy task: king is to queen as man is to what?) overwhelmingly test for lexical semantics. Extrinsic probing methods, which train classifiers on top of fixed representations, have examined morphological and syntactic content, but they measure linguistic knowledge only indirectly, through the performance of a downstream classifier.

Key Idea: Instead of asking whether embeddings know that Paris relates to France, ask whether their geometry reflects grammatical properties. The paper compares continuous embeddings with ground-truth embeddings living in a linguistically-inspired, discrete binary feature space -- the corners of an n-dimensional hypercube -- so that similarity between the two spaces can be read as grammatical, rather than lexical, content.

Why Korean affixes? (i) Korean is agglutinative, so there is largely a one-to-one mapping between affixes and meanings; (ii) its morphology is highly regular, enabling high-fidelity parsing; and (iii) Korean affixes encode at least four distinct types of information: phonological (allomorphy such as accusative -을 after a coda vs. -를 after a vowel), syntactic (place of attachment), semantic (focus markers, logical operators, case), and pragmatic (formal register, honorifics). This lets the authors run focused tests on which information types models are sensitive to.

A further obstacle is that most embedding models (Word2Vec, GloVe, even character- and syllable-level models) have no explicit notion of an affix, so there is no affix vector to inspect. The paper resolves this with transparent model distillation (Tan et al., 2017, 2018): it trains a morphologically-aware learner network to reproduce the black-box embeddings, and then inspects the learner's affix parameters as a proxy for what the original model captured. To the authors' knowledge, this is the first application of transparent model distillation for interpretation in an NLP setting.

Proposed Method

The pipeline has four stages: train ground-truth embedding models, distill each into morphologically-aware learner networks, build a discrete ground-truth feature space for affixes, and compare neighbourhoods across the continuous and discrete spaces.

1
Ground-Truth Embedding Models
Six Korean embedding models spanning three granularities are trained on a Korean WikiDump with vocabulary limited to 10,000 words and 300-dimensional embeddings (following Choi et al., 2016). Character-level: fastText (Bojanowski et al., 2017) and Naver's kor2vec (based on Kim et al., 2016). Syllable-level: the model of Choi et al. (2017). Word-level: Word2Vec Skip-gram, Word2Vec CBOW (Mikolov et al., 2013), and GloVe (Pennington et al., 2014). Other hyperparameters follow the original publications.
2
Transparent Model Distillation into Three Learner Networks
Each ground-truth model is distilled into three learner architectures that explicitly compose words from morphemes (parsed with the Komoran POS-tagger from KoNLPy). The learner iterates over the same corpus, computes its embedding ŷ for each word, and minimizes the squared distance to the ground-truth embedding y. StaffNet (Edmiston and Stratos, 2018) treats stems as vectors (combined via a BiLSTM and convex combination) and affixes as linear functions, i.e., 300×300 matrices applied iteratively to the stem. MRNN (Luong et al., 2013) is a recursive network over the binary morphological parse, computing parents as p = tanh(W[c1; c2] + b). Tree-LSTM (Tai et al., 2015) is an N-ary Tree-LSTM with N = 2 over the same binary parses. In MRNN and Tree-LSTM, affixes become 300-dimensional vectors.
3
Discrete Ground-Truth Feature Space
A hand-constructed dataset embeds 107 Korean affixes into a binary feature space {0, 1}62, where each dimension is a linguistic feature such as [+CODA] or [+HONORIFIC]. The 62 features are partitioned into four subsets: phonological ([±CODA], [±LOW]), syntactic ([+POSTPOSITION], [+CONJUNCTIVE], [+X-LIMITER], [+Z-LIMITER], [+V1]..[+V4]), semantic (46 features such as [+NOMINATIVE], [+TOPIC], [+PAST], [+CONDITIONAL], [+QUOTATIVE], [+ONLY]), and pragmatic ([±FORMAL], [+HONORIFIC], [+FAMILIAR]). Across the affixes there are 5 distinct phonological, 8 syntactic, 55 semantic and 5 pragmatic configurations. Distance in this space is Hamming distance dH.
4
Similarity in the Continuous Space
For vector-valued affixes (MRNN, Tree-LSTM), cosine similarity is used. For StaffNet's matrix-valued affixes, cosine similarity does not apply, so the paper uses subspace similarity (Mu et al., 2017): take the first N principal directions of each affix matrix, stack them into X, Y ∈ Rd×N, and score the pair by the root mean of the squared singular values of XTY, yielding a value in [0, 1]. The intuition is that similar affixes should map stems to similar subspaces. N is chosen as the minimum number of principal components explaining 25%, 50% or 75% of the average affix variance for each model.
5
Nearest-Affix Evaluation with Two Graded Scores
For each affix, the set YAff of its closest affixes under Hamming distance in the discrete space is computed (size k), and the k most similar affixes in the continuous space form ŶAff. Two scores are reported: percentage of overlap between the two sets (equivalent to an F1 measure on cluster analysis since the sets have equal cardinality), and the Hamming offset -- the average Hamming distance from the affix to the continuous-space neighbours minus the minimum possible distance. Unlike all-or-nothing analogy tests, both measures are graded. The same procedure is re-run restricting the discrete space to each feature subset (phonological, syntactic, semantic, pragmatic), and compared to a random baseline.

Experimental Results

Verifying the Distillations

Before evaluating, the paper checks that the learners faithfully reproduce the original embeddings. Average cosine similarity between ground-truth and distilled embeddings over the 10k vocabulary is high for all combinations, which is notable given that almost all of the volume of R300 is orthogonal to any given point.

ModelStaffNetMRNNTree-LSTM
kor2vec0.9710.8400.952
fastText0.9490.8240.946
Syllable0.9670.8690.979
W2V-SG0.9530.7550.920
W2V-CBOW0.9410.7860.878
GloVe0.9350.6900.875

Closest-Affix Task over All 62 Features

Percentage of overlap (higher is better) and Hamming offset (lower is better), averaged over all 107 affixes. StaffNet is shown for the subspace rank explaining 25% of variance (its best setting); the 50% and 75% settings score slightly lower.

ModelStaffNet-25% OverlapStaffNet-25% OffsetMRNN OverlapMRNN OffsetTree-LSTM OverlapTree-LSTM Offset
kor2vec12%3.3516%2.8219%2.62
fastText13%3.2318%2.5916%2.63
Syllable13%3.1817%2.8817%3.29
W2V-SG12%3.1718%2.5915%3.01
W2V-CBOW13%3.1918%2.6316%2.86
GloVe10%3.3016%2.8011%3.13
Random2%4.232%4.321%4.35

Semantic Feature Subset

The sharpest contrast between learner architectures appears when the discrete space is restricted to the semantic features (55 distinct configurations). Linear StaffNet affixes stay at the random baseline, whereas non-linear MRNN and Tree-LSTM affixes reach up to 33% overlap.

ModelStaffNet-25% OverlapStaffNet-25% OffsetMRNN OverlapMRNN OffsetTree-LSTM OverlapTree-LSTM Offset
kor2vec6%2.5423%1.8525%1.80
fastText6%2.3933%1.6524%1.72
Syllable5%2.4718%1.9619%2.04
W2V-SG5%2.4826%1.7729%1.78
W2V-CBOW6%2.5624%1.8919%2.04
GloVe5%2.4924%1.8923%1.92
Random3%2.792%2.722%2.67

Summary across Feature Subsets

Percentage-of-overlap ranges for the six embedding models versus the random baseline, by feature subset. Because the average neighbourhood size k differs greatly across subsets (e.g., avg. k = 2.21 for all features but 73.91 for pragmatic), performance must be read relative to the random baseline rather than in absolute terms.

Feature SubsetAvg. kStaffNet Overlap (models)MRNN / Tree-LSTM Overlap (models)Random Overlap
Phonological29.9329-32%28-32%26-29%
Syntactic30.9235-41%34-39%28-30%
Semantic3.254-6%18-33%2-3%
Pragmatic73.9168-71%69-72%69-71%

Why It Matters

This paper is an early, linguistically-grounded attempt to move intrinsic evaluation of word embeddings beyond lexical semantics toward grammatical information. Its contributions and implications include:

Links

Representation Learning Interpretability