Data acquisition, preprocessing and feature engineering

2.1The chain, and what each stage costs

Analog versus digital sensors. An analog sensor delivers a continuous voltage and requires the whole chain above; you own the anti-alias filter, the reference, and the noise budget. A digital sensor (an I²S MEMS microphone, an I²C IMU) has the converter on-die and hands you samples over a bus — convenient, but the conversion decisions have been made for you and are visible only in the datasheet. The important point is that "digital sensor" does not mean "no signal processing"; it means the processing happened somewhere you cannot change, so you must read its specification: sensitivity, acoustic overload point, SNR, and the on-chip decimation filter's group delay.

2.2The sensors you will meet

A sensor (more precisely, a transducer plus its conditioning electronics) converts a physical quantity — pressure, acceleration, light, voltage across skin — into an electrical signal. Most sensors used in embedded ML today are MEMS devices (micro-electro-mechanical systems): microscopic mechanical structures etched in silicon, packaged with their own amplifier and often their own analog-to-digital converter, which deliver digital samples over a serial bus. For the ML engineer, four properties of a sensor matter more than its physics: its sampling rate, its resolution and noise floor, its interface, and its power draw while sampling.

Table 2.1 — Sensors in embedded ML. Order-of-magnitude ranges across common parts; the active-power column is the sensor alone. Note that a gyroscope costs one to two orders of magnitude more power than an accelerometer — a classic reason to infer orientation from the accelerometer alone when the application allows it.

Two practical lessons recur. First, many sensors can do part of the processing themselves: modern IMUs contain FIFO buffers, step counters and even small programmable state machines or decision-tree engines, so the main MCU can sleep until something interesting happens — the cascade of Session 1 implemented inside the sensor. Second, the sensor's configuration (range, rate, filter) is part of the model's input specification: a classifier trained on data captured at ±4 g and 100 Hz will fail silently on a device configured for ±16 g and 50 Hz. Record the configuration with the data.

2.3Signals in time and frequency — a primer

A signal here is a sequence of numbers indexed by time, x[n] = x(n·T_s), where T_s is the sampling interval and f_s = 1/T_s the sampling rate. You can describe it in two equivalent ways: by its values over time, or by how much of each frequency it contains. The second description is the key to almost everything in this session.

The bridge is Fourier's observation that any reasonable signal is a sum of sinusoids. A sinusoid A·sin(2π f t + φ) has three properties: amplitude A, frequency f (cycles per second, Hz) and phase φ. The spectrum of a signal lists the amplitude (and phase) of each frequency it contains.

Why should an ML engineer care? Because most physical processes are easier to recognise in frequency than in time. A bearing fault produces vibration at a characteristic frequency; a spoken vowel is a pattern of resonances; walking is a periodic motion at about 2 Hz. And because the operations of the sensing chain — filtering, sampling, windowing — have simple descriptions in the frequency domain and complicated ones in time.

2.4Sampling and aliasing, stated precisely

Sampling theorem. A signal whose spectrum is zero outside |f| < B is completely determined by samples taken at rate f_s > 2B, and is reconstructed by sinc interpolation. Sampling in time multiplies by an impulse train, which convolves the spectrum with an impulse train of spacing f_s: the spectrum is periodically replicated. If f_s < 2B, adjacent replicas overlap and add. That overlap is aliasing, and it is an addition, not an occlusion — which is why it is irreversible.

A component at frequency f appears after sampling at the folded frequency

so a 30 kHz interferer sampled at 16 kHz lands at 2 kHz, squarely inside a speech band, and is now indistinguishable from real speech energy.

2.5Quantization noise, SQNR and ENOB depth

Dither. The independence assumption fails for very small or very periodic signals, where the error becomes correlated with the signal and appears as harmonic distortion — audible as tones rather than hiss. Adding a small amount of noise (about 1 LSB rms) before quantization decorrelates the error, trading a slightly higher noise floor for the removal of structured artefacts. This matters for embedded audio because structured artefacts are exactly what a spectrogram-based classifier will latch onto.

Averaging and oversampling. Averaging M independent samples of a stationary signal reduces noise power by M, i.e. SNR improves by 10 log₁₀ M dB — about 3 dB per doubling, or half a bit. Getting one extra effective bit therefore costs 4× the samples. This is the exchange rate between analog quality and digital effort, and it is the basis of sigma-delta converters, which deliberately oversample by 64–256× and shape the quantization noise out of the band of interest.

2.6Noise budget and filtering

Sources, in the order they usually dominate on a real board: power-supply and switching noise coupled through the reference; electromagnetic interference at mains frequency and its harmonics; thermal (Johnson) noise, white, proportional to √(4kTRB); 1/f (flicker) noise, dominant at low frequencies and the reason DC-coupled measurements are hard; quantization noise, usually the smallest of the five. Note the ordering: students are taught quantization noise and then find that it is rarely the limiting term.

FIR versus IIR on an embedded target

Table 2.3 — Choosing a filter structure. For a given magnitude specification an IIR filter is typically an order of magnitude cheaper than an FIR one; you pay in phase linearity and numerical fragility.

The moving average is an FIR filter with impulse response 1/M over M taps, and its frequency response is a Dirichlet (periodic sinc) kernel: a wide main lobe and first sidelobe only about 13 dB down. It is cheap and it is a poor low-pass filter, and it is worth looking at its magnitude response once, to stop treating "smoothing" as free. The moving median, by contrast, is non-linear: it removes impulsive outliers without smearing edges, has no frequency response at all, and costs a sort or a running-order-statistic structure — often the right choice for IMU spike removal and the wrong choice for anything you will later transform.

Fixed-point implementation. On MCUs without an FPU, filters run in Q-format fixed point. Three rules: accumulate in a wider type than you multiply in (Q15 × Q15 → Q30 in a 32-bit accumulator); scale coefficients so intermediate results cannot overflow, or use saturating arithmetic; and implement high-order IIR filters as cascaded second-order sections, because direct-form high-order implementations are numerically catastrophic. CMSIS-DSP's biquad functions exist for exactly these reasons.

2.7Sampling rate and resolution are energy decisions

Sampling rate propagates multiplicatively through the whole downstream cost: ADC energy per conversion × rate, filter MACs per sample × rate, frames per second × feature cost, inferences per second × model cost. Halving f_s where the application allows it is usually the single largest energy saving available, and it is free of accuracy loss if the discarded band carries no class-discriminative information — a claim that should be tested empirically, not assumed. For keyword spotting, 16 kHz is conventional and 8 kHz is often sufficient; for bearing-fault detection the diagnostic energy may sit above 10 kHz and the same reasoning gives the opposite answer.

Oversample-and-decimate is the standard escape from expensive analog design: sample at R·f_s with a gentle analog filter, apply a sharp digital low-pass, then keep every R-th sample. Two efficiency notes for embedded targets: the decimating filter should be implemented as a polyphase structure so it only computes the outputs that survive (an R× saving), and cascaded integrator-comb (CIC) filters achieve large decimation ratios with no multipliers at all, which is why they appear inside PDM microphone front ends.

2.8Datasets, labels and preprocessing

The signal chain delivers samples; a learning algorithm needs a dataset: many examples, each paired with a label, that represent the conditions the device will meet in the field. For embedded sensing problems the dataset is usually the most expensive and least reusable part of the project, and the most common cause of disappointing field performance.

Collection and labelling

• Coverage beats volume. Record across the variation the device will see: different users, mounting positions, devices of the same model (sensor gains differ), environments, background activities. A thousand recordings from one person are worth less than a hundred from fifty.
• Label granularity is a design decision. A recording may be labelled as a whole ("this minute contains a fall"), per window, or with exact event boundaries. Coarse labels are cheap; fine labels are needed to train detectors with accurate onsets. Weakly labelled learning is an active research area precisely because fine labels are so expensive.
• Rare events and class imbalance. The events you care about — faults, falls, arrhythmias, a wake word — are usually rare. Plan for it: collect targeted examples, use a background ("none of the above") class that is genuinely diverse, and evaluate with metrics that are not fooled by imbalance (Session 3).
• Public datasets. Speech Commands (keyword spotting), ESC-50 and UrbanSound8K (environmental audio), UCI-HAR and PAMAP2 (activity recognition), MIT-BIH (arrhythmia), MIMII and ToyADMOS (machine sound anomalies), Visual Wake Words (person detection). They are ideal for comparison and teaching, and dangerous as a proxy for your own deployment conditions.

Cleaning and preprocessing

Before features, the raw data usually needs: synchronisation of streams sampled by different clocks; resampling to the rate the deployed device will use; handling of gaps (dropped packets, sensor saturation) — by interpolation, masking or discarding windows; outlier and artefact removal (the median filter above); and calibration (offset and gain per sensor). Every one of these steps must also exist, identically, in the firmware — a discrepancy between the training pipeline in Python and the deployed pipeline in C is one of the most common causes of embedded models that "worked in the notebook".

Normalisation

Most learning algorithms assume features on comparable scales. Standardisation (z-scoring) subtracts the training-set mean and divides by the training-set standard deviation of each feature; min–max scaling maps the training range to [0, 1]. Decision trees are indifferent to scaling; neural networks, SVMs, k-nearest neighbours and anything trained by gradient descent are not (the reason is visible in the gradient-descent figure of Session 3). On an embedded device the normalisation is two constants per feature, applied in the firmware — and, conveniently, it can be folded into the first layer's weights and the quantization parameters (Session 3), so it often costs nothing at run time.

Augmentation

Data augmentation creates plausible new training examples from existing ones: time shifts, added background noise at various signal-to-noise ratios, gain changes and room reverberation for audio; small rotations of the sensor frame, time warping and amplitude scaling for IMU data. It is free at inference time and often the cheapest accuracy improvement available to a small model, because small models are more sensitive to the gap between training and field conditions. SpecAugment-style masking of time and frequency bands is standard for spectrogram inputs.

2.9Framing, windowing and segmentation

Models consume frames, not streams. A frame length L with hop H gives an overlap of 1 − H/L and a frame rate of f_s/H. Three consequences: the frame rate sets the inference rate and therefore the energy; the frame length sets the lowest frequency that can be resolved (f_s/L); and the hop sets the temporal precision of an event's onset. Overlap is not free — 50 % overlap doubles the number of inferences — so on an energy-limited device it is a parameter to be justified, not a default. §2.12 takes this up properly with the time–frequency uncertainty relation.

For event-like data the fixed sliding window is not the only option. Event-triggered segmentation starts a window when a cheap detector (an energy threshold, a step detector in the IMU) fires, which aligns windows with the events and saves energy between them. Labels must be assigned per window — by majority vote over the samples in it, by "any event present", or by the label at the window centre — and that choice changes what the model learns about onsets. Write it down.

2.10Three questions to ask of any representation

1. What does it discard, and is the discarded part class-discriminative? Mel warping discards fine high-frequency resolution — irrelevant for speech, potentially fatal for ultrasonic fault detection.
2. What invariances does it build in? The magnitude spectrogram discards phase and thereby buys invariance to time shift within a frame. If your task depends on inter-channel phase (direction of arrival), you have just deleted the signal.
3. What does it cost, and against what baseline? Compare the front end's MACs and memory with the model's. If the front end costs 40 % of the budget, a learned front end operating on raw samples may be cheaper overall.

2.11Time-domain features

The cheapest features are computed directly on the samples of a window, with one pass and no transform. For a window x[0…N−1]:

plus higher-order statistics (skewness, kurtosis), percentiles, and counts of threshold crossings. Each costs O(N) operations — tens of microseconds on an MCU for a typical window — and many can be updated incrementally as each sample arrives, so the window never needs to be stored at all.

These features are what the first stage of a cascade is built from: a voice-activity detector, a "something is moving" trigger for an IMU, or an "unusual vibration energy" alarm. They are also more powerful than they look for slowly varying physical signals, and a sensible baseline for any new task: if a decision tree on ten time-domain statistics already reaches the target accuracy, the project does not need a neural network.

2.12The short-time Fourier transform depth

The Fourier transform assumes stationarity over its whole support; speech, vibration and gesture signals are not stationary. The STFT applies the transform to short windowed segments in which stationarity is approximately true:

with window w of length L and hop H. The spectrogram is |X[m,k]|².

Windows and leakage

Truncating with a rectangular window multiplies in time, hence convolves in frequency with a Dirichlet kernel whose first sidelobe is only ≈13 dB down. A strong component therefore leaks into neighbouring bins and can bury a weak one. Tapered windows trade main-lobe width (resolution) for sidelobe suppression (dynamic range):

Table 2.4 — Window trade-offs. Widths are approximate, in units of the DFT bin spacing f_s/L. Choosing a window is choosing where on the resolution/dynamic-range curve your task sits.

The uncertainty relation, and what it means operationally

Frequency resolution is Δf ≈ f_s/L and time resolution is Δt = L/f_s, so Δt·Δf ≈ 1 — a bound, not a tuning knob (for a Gaussian window the exact statement is σ_tσ_f ≥ 1/4π). You cannot resolve a 20 ms transient and a 10 Hz frequency difference in the same analysis. Increasing overlap increases the frame rate and thus the smoothness of the picture, but it does not improve Δt; the smearing is set by L. Students conflate these constantly.

2.13From spectrogram to mel and MFCC

Human auditory frequency resolution is approximately logarithmic above ~1 kHz. The mel scale encodes this; the most commonly used form is

A mel filterbank places B triangular filters equally spaced on the mel axis and applies them to the power spectrum, reducing, say, 257 FFT bins to 40 mel bands — a 6× dimensionality reduction that is nearly free of task-relevant information for speech. The classical MFCC pipeline then continues:

Why each step exists. Pre-emphasis (y[n] = x[n] − αx[n−1], α ≈ 0.97) flattens the roughly −6 dB/octave spectral tilt of voiced speech so that high-frequency detail is not numerically swamped. Log compression converts multiplicative channel effects into additive offsets, which is what makes cepstral mean normalisation a valid channel-compensation method. DCT-II approximates the Karhunen–Loève transform for this class of signals, decorrelating the coefficients; keeping the first 12–13 discards the fine spectral detail that encodes pitch. Liftering rescales coefficients so their dynamic ranges are comparable.

Deltas. First and second time derivatives (Δ, ΔΔ), computed by regression over ±2 frames, add dynamic information to an otherwise static frame representation. For a CNN or an RNN with temporal context they are largely redundant, since the model can compute them; for an SVM or GMM over single frames they are essential. This is a good illustration of the general rule that feature engineering compensates for what the model class cannot learn.

2.14IMU and image features

IMU (accelerometer / gyroscope). A well-tested feature set for human-activity recognition, per axis per window: mean, standard deviation, min, max, interquartile range, root-mean-square, skewness, kurtosis, zero-crossing rate, signal-magnitude area, correlation between axes, and the norm √(a_x²+a_y²+a_z²), which is rotation-invariant and therefore robust to how the device is worn. Frequency-domain additions: dominant frequency, spectral entropy (flatness of the spectrum, distinguishing periodic walking from aperiodic fidgeting), band energies, and autocorrelation peak lag for cadence. Jerk (the derivative of acceleration) separates smooth from abrupt motions. Note that on a 50 Hz IMU a 2.56 s window is 128 samples — an FFT of that size is trivial, so the choice of time- versus frequency-domain features is about statistics, not cost.

Images. Handcrafted descriptors — edges, corners, colour histograms, HOG, LBP, Haar features over an integral image — were the state of the art before 2012 and remain competitive under extreme constraints, where the alternative model would have to be very small. In current practice a learned front end (a few convolutional layers of a MobileNet-class network) dominates whenever the compute exists, and the interesting question for this course is exactly where the crossover sits on a given device.

Table 2.5 — Order-of-magnitude front-end costs, for building the habit of comparing front end against model. The comparison flips for very small models and for high sample rates: measure, do not assume.

2.15Feature selection depth

Three families, in increasing cost and decreasing generality:

• Filter methods score each feature against the label independently of any model. Pearson correlation ρ = cov(X,Y)/(σ_Xσ_Y) captures only linear dependence and will assign zero to a perfect quadratic relationship. Mutual information, I(X;Y) = H(Y) − H(Y|X), captures any dependence but must be estimated — typically by k-nearest-neighbour estimators — and estimation error grows quickly with dimension. Filters are cheap and ignore redundancy: two perfectly correlated informative features both score highly and both get selected.

• Wrapper methods search over subsets using the model itself as the scoring function. Recursive feature elimination fits the model, ranks features by an importance measure, removes the weakest, and repeats. Expensive but accounts for interactions and redundancy.
• Embedded methods select during fitting. Lasso minimises ‖y − Xβ‖² + λ‖β‖₁; the L1 penalty's corner geometry drives coefficients exactly to zero, so the fit and the selection are one problem. Tree ensembles give impurity- or permutation-based importances — prefer permutation importance, since impurity importance is biased toward high-cardinality features.

2.16Dimensionality reduction

LDA is supervised: it maximises the Fisher criterion w^TS_Bw / w^TS_Ww, the ratio of between-class to within-class scatter. Its output is capped at C−1 dimensions for C classes — a hard structural limit worth remembering — and it assumes roughly Gaussian, equal-covariance classes.

t-SNE and UMAP are non-linear neighbour-embedding methods for visualisation. Both optimise a notion of local neighbourhood preservation; both distort global distances; t-SNE has no natural out-of-sample extension at all. They are excellent for inspecting whether your classes separate before you spend a week training, and they must never appear in a deployed pipeline. Beware of reading cluster sizes and inter-cluster distances in a t-SNE plot: neither is meaningful.

Random projection deserves more attention than it gets in embedded work. The Johnson–Lindenstrauss lemma guarantees that projecting onto O(log n / ε²) random directions preserves all pairwise distances to within 1±ε. With a sparse or ±1 projection matrix generated from a stored seed, the projection costs additions only and needs no stored matrix — an unusually good fit for a microcontroller.

PCA versus LDA, and t-SNE, on pictures

The two figures below make the comparison of this section concrete. In the first, the direction of greatest variance is exactly the wrong one to keep for classification; LDA, which uses the labels, finds the one that separates the classes. Both are a fixed matrix multiply at inference time, with parameters estimated on the training split only.

The second figure shows the same contrast between a linear projection and a non-linear embedding on real data. Use the t-SNE view to understand a dataset — whether classes form clusters, which examples are mislabelled, whether one subject's recordings sit apart from the rest — and never as a stage of the deployed pipeline.

Before Session 3

• Y. Zhang, N. Suda, L. Lai, V. Chandra, "Hello Edge: Keyword Spotting on Microcontrollers," 2017. arXiv:1711.07128.A complete embedded pipeline — MFCC front end, several model families, memory and operation budgets — in twelve pages. Session 3 uses its numbers.
• If you have no ML background: work through the primer sections at the start of Session 3 before the lecture.

Discussion questions

1. A colleague proposes to sample an accelerometer at 25 Hz "because human motion is below 10 Hz". What do you need to know about the sensor's internal filter before agreeing?
2. Name a task for which discarding phase (taking the magnitude spectrogram) destroys the information the model needs.
3. When is a learned front end (a 1-D convolution on raw samples) likely to be cheaper overall than an FFT plus mel filterbank?
4. Your test accuracy drops from 97 % to 81 % when you change from a random split to a subject-wise split. What have you learned, and what do you report?

Exercises

1. A 12-bit ADC digitises a sensor whose useful signal spans 20 % of the full-scale range. What is the effective SQNR for that signal, and how many bits are effectively "used"?
2. Compute the number of frames, the frequency resolution and the output size of a 512-point STFT with hop 160 on one second of 16 kHz audio. How do they change with a 1024-point frame?
3. An IMU samples at 50 Hz. You need to recognise gestures lasting 0.6–1.5 s with a decision latency below 1 s. Propose window length, hop and three features, and estimate the compute per second.
4. Derive the cost in MACs of computing 40 log-mel energies from a 512-point power spectrum when the filterbank is stored sparsely.

Further reading

• A. V. Oppenheim and R. W. Schafer, Discrete-Time Signal Processing, 3rd ed., Pearson, 2010.The reference for sampling, filtering and the DFT.
• P. Warden, "Speech Commands," 2018. arXiv:1804.03209.Dataset design for embedded audio.
• D. S. Park et al., "SpecAugment," Interspeech 2019. arXiv:1904.08779.The standard augmentation for spectrogram inputs.
• Q. Kong et al., "PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition," IEEE/ACM TASLP 2020. arXiv:1912.10211.How far learned audio representations go when compute is plentiful — the opposite end of the spectrum from this course.