
Acoustic Echo Cancellation in python3-sipsimple
===============================================

This document describes how software echo cancellation works in sipsimple,
why it degraded after the 2021 pjsip migration, what was changed in
September 2026 (pjsip 2.17 patch series, 09_aec.patch, plus sipsimple core)
and how to measure it.


1. Signal path
--------------

  AudioMixer (sipsimple/core/_core.sound.pxi)
    pjmedia_snd_port_create2(ec_enabled=1, ec_tail_ms=tail_length,
                             ec_options=_ec_options)
      sound_port.c   play_cb -> pjmedia_echo_playback()   (output thread)
                     rec_cb  -> pjmedia_echo_capture()    (input thread)
        echo_common.c  latency list + delay buffer (WSOLA) that pairs each
                       captured frame with the frame played latency_ms
                       earlier (latency_ms = 3/4 of the device output
                       latency, at least one 20 ms frame)
          echo_webrtc.c  WebRTC AEC (the pre-AEC3 "AEC2" core) + optional
                         WebRTC noise suppressor

  The device echo canceller (macOS VoiceProcessingIO) is disabled by patch
  05 (was 16_mac_audio_device_fix), so the software canceller is always used.

  Settings:
    audio.echo_canceller.enabled      (Blink forces 32 kHz when enabled,
                                        48 kHz and EC off when disabled)
    audio.echo_canceller.tail_length  default 2.  Despite the name this is
                                      passed to WebRtcAec_Process() as the
                                      reported sound card delay.  The value
                                      was tuned for the 2013 wrapper.  Leave
                                      it: over-estimating the delay by
                                      100 ms or more breaks cancellation
                                      completely (see section 5).


2. History (from the darcs repository)
---------------------------------------

  2011-09  "Added WebRTC AEC" (python-sipsimple / Blink Pro 1.3).
  2013-06  pjsip 2.x r4531 in deps/pjsip.  Saul Ibarra adds
           pjmedia/src/pjmedia/echo_webrtc_aec.c with its own WebRTC copy,
           compiled from pjmedia/build/os-auto.mak.in
           ("PJMEDIA_OBJS += echo_webrtc_aec.o").
  2013-07  Tuning by Saul Ibarra and Dan Pascu:
             - "Fixed echo cancelling when using 32kHz sample rate"
               (QMF band split, WebRtcSpl_AnalysisQMF/SynthesisQMF)
             - "Fixed resetting AEC state" (reset deferred to the
               processing thread to avoid races)
             - "Add AGC and high pass filter to audio processing"
             - "Increased echo canceller aggressiveness" (nlpMode 2)
             - "Disable skewMode", "Improved echo canceller reset"
               (mute 500 ms after a reset)
             - "Use optimal value for the echo canceller tail length" (2)
  2021-01  2to3 port (python3-sipsimple), still the old in-tree pjsip.
  2021-02..05  Migration to the pjproject 2.10 tarball + patch series
           ("Added migration scripts for pjsip 2.10", "Removed obsolete
           pjsip files", "Refactor initial pjsip setup").
           echo_webrtc_aec.c is carried inside 001_pjsip_210.patch, but
           the os-auto.mak.in line that compiled it and the old WebRTC
           sources are not.  From here on every build links pjsip's
           upstream echo_webrtc.c.  Nothing fails, so this went unnoticed.
  2023-11  "Added echo canceller to Raspbian build".
  2025-02  pjsip 2.11 / Apple Silicon (neon WebRTC build).
  2026-05  pjsip 2.12, 001_pjsip_210.patch split into 01..12; the dead
           file becomes 09_aec.patch.
  2026-06  pjsip 2.17 series; 09_aec left as "TODO, audio quality
           regression vs 2.12".  In reality 2.12 and 2.17 used the same
           upstream code (echo_webrtc.c, echo_common.c, the WebRTC library
           and the EC parts of sound_port.c are identical apart from
           whitespace).

  Verification of the above: strings(1) on the shipped _core*.so of the
  tarball based builds (Distribution/staging, App Store 2026-05-09) show
  echo_webrtc.o and the upstream log messages ("WebRTC AEC successfully
  created with options %d"), never echo_webrtc_aec.o.

  So "echo cancellation worked well in the past" refers to the 2013
  wrapper that was in use until the 2021 migration.


3. Problems found in the upstream wrapper as used by sipsimple
--------------------------------------------------------------

  a) Wrong processing at 32 kHz and 48 kHz (pjsip issue #2404, wontfix).
     The AEC core must be given the 0-8 kHz band split off with the QMF
     filter bank plus the upper band(s).  echo_webrtc.c passes 160 full
     band samples with num_bands=1.  The core then processes 32 kHz audio
     as if it was 16 kHz: half the filter length, half the delay estimator
     range, mis-scaled suppression.  aec_core.c asserts on it; release
     builds use -DNDEBUG so nobody noticed.  Result: slow and unstable
     cancellation that comes and goes.  This is the "sometimes it does not
     work" symptom.

  b) Race conditions on reset.  AudioStream.start() and update_direction()
     call AudioMixer.reset_ec() -> pjmedia_snd_port_reset_ec_state() from
     the Python thread, and sound_port.c (patch 08) resets the canceller
     when it resumes after 5 s without playback.  pjmedia_echo_reset()
     rearranges the latency lists and re-initializes the WebRTC core
     without any locking while the CoreAudio/ALSA threads use them.
     The trace of a real call (blink/logs/pjsip_trace.txt, 2026-08-31
     11:53:02) shows three resets within 30 ms at call start, the last one
     2 ms after the latency buffer had been primed by the audio threads.  A stress test
     crashes the unpatched code in 4 of 25 runs.

  c) Slow recovery.  The WebRTC core ignores its own delay estimate for
     the first 15 s after every init or reset.  Since pjsip does not report
     the real system delay, any echo path that is not already aligned
     (Bluetooth or external speakers, HDMI monitors, anything above
     roughly 60 ms) goes uncancelled for 15 s at the start of each call and
     again after each reset, hold/unhold or 5 s of silence from the remote
     party.

  d) No noise suppressor, no high pass filter.  sipsimple passed
     ec_options=0.  The 2013 wrapper always ran NS and an 80 Hz high pass
     filter, and Blink's preference text still promises noise reduction.


4. Changes
----------

  deps/patches/2.17/09_aec.patch (pjsip)

    echo_webrtc.c
      - 10 ms block processing at the core rate.
      - 32 kHz: QMF analysis of near and far end, core with num_bands=2,
        QMF synthesis of the output.
      - Rates the core does not support natively (48 kHz, 44.1 kHz, ...):
        resample to 32 kHz (16 kHz below 16 kHz) and back.
      - 80 Hz high pass filter on the capture, noise suppressor band aware.
      - AECM (mobile) path unchanged.

    echo_common.c / echo.h
      - pjmedia_echo_reset() is deferred: it sets a flag and the capture
        thread applies it before the next frame.  The latency list is
        handed between threads with lat_ready, like the existing prefetch
        logic, so the audio path takes no lock.
      - New pjmedia_echo_resync(): flush and re-prime the latency and delay
        buffers but keep the learned echo path.

    echo_webrtc.c statistics
      - The capture thread snapshots the WebRTC delay metrics and
        ERL/ERLE/NLP metrics once per second; webrtc_aec_get_stat() copies
        the snapshot (seqlock), so pjmedia_snd_port_get_ec_stat() is safe
        from any thread.  Upstream called the metric functions directly from
        the caller's thread, racing the capture thread.

    sound_port.c / sound_port.h
      - EC resume after inactivity uses resync instead of reset.
      - New pjmedia_snd_port_resync_ec_state()
        (#define PJMEDIA_HAS_SND_PORT_RESYNC_EC 1).

    third_party/webrtc/.../aec/aec_core.c
      - Delay correction starts after 200 frames (2 s) instead of 1500
        (15 s).  Override with -DWEBRTC_AEC_DELAY_CORRECTION_START=N.

    third_party/build/webrtc/Makefile, third_party/webrtc/CMakeLists.txt
      - compile common_audio/signal_processing/splitting_filter.c.

  sipsimple/core

    _core.pxd              ec_options field of pjmedia_snd_port_param,
                           pjmedia_echo_flag constants.
    _core.sound.pxi        AudioMixer passes
                           _ec_options = PJMEDIA_ECHO_USE_NOISE_SUPPRESSOR |
                                         PJMEDIA_ECHO_AGGRESSIVENESS_AGGRESSIVE
                           (the 2013 tuning).  New AudioMixer.resync_ec();
                           it falls back to a full reset when built against
                           a pjsip tree without 09_aec.
    _core.mediatransport.pxi
                           AudioStream start and direction change call
                           resync_ec() instead of reset_ec(), so what the
                           canceller learned (e.g. while ringback played) is
                           kept.  reset_ec() is still available for callers
                           that know the acoustic path changed.
                           New AudioMixer.ec_statistics property (dict or
                           None), see section 7.

  Blink

    SessionController.py   _NH_SIPSessionDidStart no longer calls
                           mixer.reset_ec(); it discarded what the canceller
                           had learned during ringback, right after sipsimple
                           had resynchronized it.
    AudioController.py     updateEchoCancellerStatistics(): every 10 s while
                           connected logs "Echo canceller: <info>" (debug
                           log) and at the end of the call "Echo canceller at
                           end of call: <info>" (info log).  Both also post
                           BlinkAudioEchoCancellerStatistics.  Silently
                           skipped with an older sipsimple without
                           ec_statistics.
    DebugWindow.py         RTP tab shows every statistics sample ("Audio echo
                           canceller to <uri>: ...") regardless of debug
                           mode.  A line turns red after 5 s of processing
                           when ERL < 12 dB and ERLE < 10 dB ("echo may be
                           audible").

  A new sound device (device change, set_sound_devices) creates a new
  canceller, so no explicit reset is needed there.


5. Measurements
---------------

  Harness: deps/patches/2.17/aec-harness (see section 6).  Speech from the
  pjproject test vectors, simulated room (direct path + early reflections +
  exponential diffuse tail), noise floor -75 dBFS, 8 s of double talk from
  24 to 32 s.  Same parameters as sipsimple: 20 ms frames, tail 2,
  latency 20 ms.  Numbers are ERLE in dB per 2 s window (higher is better,
  below ~10 dB the echo is clearly audible).

  32 kHz, 60 ms delay, 150 ms reverb
    before  8 11 11  5 19 35 35 34 35 35 35 35 DT DT DT DT 12 35 34 34
    after  14 27 28 30 29 29 29 28 29 28 29 28 DT DT DT DT 27 28 28 28

  32 kHz, 150 ms delay, 300 ms reverb
    before  3  2  3  1 18 32 30 31 14 33 12  8 DT DT DT DT  8 14  9  8
    after   3  5 28 29 29 28 29 28 28 28 28 27 DT DT DT DT 26 28 27 28

  48 kHz, 150 ms delay, 300 ms reverb
    before  3  2  4 14 18 34 12  9  6 17 13 20 DT DT DT DT  8 20 11 22
    after   7 17 30 32 31 31 31 31 31 31 31 31 DT DT DT DT 29 30 30 30

  16 kHz, 150 ms delay, 300 ms reverb (before: correct band handling but
  the 15 s delay correction start)
    before  3  5  5  3  4  2  1  3 26 25 25 24 DT DT DT DT 23 23 23 23
    after   3 13 26 27 26 26 27 25 27 24 27 25 DT DT DT DT 25 25 26 24

  Options at 32 kHz, 60 ms (steady state ERLE / near end attenuation during
  double talk / near end attenuation with a headset, i.e. no echo at all):
    none                      29 dB /  7.7 dB / ~1 dB
    aggressive                29 dB /  9.8 dB / ~2 dB  (better first seconds)
    noise suppressor          35 dB /  9.5 dB / ~2 dB  (more while NS learns)
    NS + aggressive (default) 35 dB / 11.6 dB / ~3 dB
  The default trades a few dB of near end level during double talk for
  less residual echo, which is what the 2013 tuning chose.  Change
  _ec_options in _core.sound.pxi to revisit.

  Reported delay (tail_length) sensitivity at 32 kHz with the fix:
    actual  30 ms: tail 2 / 60 good, tail 150 / 300 never converge
    actual  60 ms: tail 2 / 60 good, tail 150 ~25 dB, tail 300 never
    actual 150 ms: tail 2 converges in 4 s, tail 60 in 2-4 s, tail 150
                   instantly, tail 300 stays around 17 dB
  => keep the default of 2.  Only raise it for a known fixed setup.

  Regression checks for the earlier delay correction (final build): far
  end silent then at -35 dB for the first 10 s (converges 8-10 s, as soon
  as the far end is audible), far end muted for 10 s mid-call (back to
  30-50 dB right after), headset without echo (near end passes, see the
  table above).  No regressions found.

  Reset stress test (playback thread, capture thread, a third thread
  alternating reset/resync as fast as possible for 20 s, then 20 s clean):
    before, 16 kHz: segfault 4/25; other runs never recover
    after, 16/32/48 kHz: 0/75 failures, 28-29 dB again within 2-4 s
  Occasional short dips in the last 20 s also happen with the reset thread
  disabled: they come from the artificial thread pacing (the delay buffer
  inserts/drops a frame) and are unrelated to the reset code.


6. Harness
----------

  deps/patches/2.17/aec-harness/

    ec_run.c      offline runner: feeds far/mic raw files through
                  pjmedia_echo_playback()/pjmedia_echo_capture() like
                  sound_port.c does.
                  ec_run RATE far.raw mic.raw out.raw OPTIONS TAIL [LATENCY]
                  [RESET_AT_FRAME ...]
                  EC_STATS=1 prints pjmedia_echo_get_stat() every 5 s.
    stress.c      three thread reset/resync/get_stat stress test (build
                  with -DHAVE_RESYNC for a tree with 09_aec).
    gen.py        scenario generator: gen.py RATE DELAY_MS RT_MS PREFIX [1=DT]
    gen_var.py    quiet start / mid-call mute / headset (no echo) variants.
    evalec.py     ERLE summary, timeline.py ERLE per 2 s window.
    build.sh      build.sh <built pjsip tree> <out> [source.c] [cflags]
    run_all.sh    runs the table above.

  Requirements: python3 with numpy and scipy.  A pjsip tree configured and
  built with at least pjlib, pjlib-util, third_party and pjmedia.  Compile
  WebRTC with -DNDEBUG (as setup_pjsip.py does), otherwise the unpatched
  code aborts at 32/48 kHz on the num_bands assert.  Set PJSIP_TREE to a
  pjproject source tree for the speech vectors.

    cd deps/pjsip
    ./configure --disable-video --disable-sound --disable-ssl \
                CFLAGS="-DNDEBUG -O2 -g"
    make -C pjlib/build && make -C pjlib-util/build && \
    make -C third_party/build && make -C pjmedia/build pjmedia
    cd ../patches/2.17/aec-harness
    ./build.sh ../../../pjsip ec_run
    PJSIP_TREE=../../../pjsip ./run_all.sh ./ec_run

  (On the full series tree pjmedia/build/Makefile also lists
  transport_zrtp.o and the other patched objects; build the normal way or
  drop them for a quick harness build.)


7. Checking a live call
-----------------------

  With PJSIP log level 4-5 in Blink, a correct setup logs:

    echo_webrtc.c WebRTC AEC successfully created with options 12416,
      clock rate 32000, core rate 32000, 2 band(s), noise suppressor
    ec0x... WebRTC AEC created, clock_rate=32000, ... latency=20 ms
    sound_port.c EC resync requested          (stream start, hold/unhold)
    ec0x... EC resynchronized                 (applied by capture thread)
    sound_port.c EC suspended because of inactivity / EC activated

  "EC reset" (full reset) should only show up when AudioMixer.reset_ec()
  is called explicitly.  "resampling" in the first line means a device
  rate other than 8/16/32 kHz.

  Blink debug window, RTP tab (every 10 s during the call):

    2026-09-11 14:02:10 Audio echo canceller to sip:alice@example.com:
      delay=20ms std=0ms poor=0.00 ERL=6dB ERLE=23dB (now 26dB) NLP=8dB
      rate=32000/32000 9s

  Blink activity log (statistics, section 4):

    Echo canceller: delay=20ms std=0ms poor=0.00 ERL=6dB ERLE=23dB
      (now 26dB) NLP=8dB rate=32000/32000 9s
    Echo canceller at end of call: ...

  AudioMixer.ec_statistics keys: backend, delay, delay_std,
  poor_delay_fraction, erl, erle, duration (s since last reset), info.
  Values that are not available are None (shown as "n/a" in info, e.g.
  ERL before WebRTC has measured it); the property itself is None during
  the first second after creation or reset.

  Reading the numbers (from the harness, section 5):

    delay     residual delay the AEC aligned to after its own buffer
              shift, not the acoustic delay.  Stable (std 0) is good.
    poor      share of delay estimates outside the adaptive filter.  Above
              ~0.3 the echo path is not aligned yet (first seconds after a
              reset on a long echo path, e.g. 0.65 at 150 ms before the
              delay correction kicks in, 0.00 once aligned).
    ERL       speaker-to-mic coupling.  Low (0-10 dB) means loud echo at
              the mic (laptop speakers); 20+ dB means little echo, e.g. a
              headset.  With high ERL, delay and poor are meaningless.
    ERLE      removed by the linear filter (the noise suppressor adds on
              top).  Converged calls show 20-45 dB; below ~10 dB with low
              ERL means audible echo.  "now" is the instantaneous value and
              drops during double talk.
    NLP       extra attenuation by the non-linear processor.

  So a call with echo complaints typically shows low ERL together with
  low ERLE and a high poor fraction that does not go down.


8. Known limitations and possible follow-ups
--------------------------------------------

  - AGC: the 2013 wrapper also ran the WebRTC analog AGC fed with the far
    end.  The WebRTC copy bundled with pjsip has no AGC module, so it is
    not included.
  - The reported delay is still the fixed tail_length.  A better initial
    estimate could come from the device's input + output latency
    (pjmedia_aud_param) minus the latency buffer, which would give instant
    convergence on long echo paths.  Over-estimates are harmful, so this
    needs field data first.
  - Separate input and output devices (e.g. MacBook mic + Bluetooth
    speaker) run on different clocks.  The WSOLA delay buffer compensates
    by stretching the reference, which costs cancellation after each
    adjustment.  PJMEDIA_ECHO_USE_SIMPLE_FIFO did slightly better in the
    stress test but was not changed without field testing.
  - WebRTC AEC3 (--enable-libwebrtc-aec3, PJMEDIA_ECHO_WEBRTC_AEC3) handles
    16/32/48 kHz natively and has AGC2.  It is not built today (arm64 macOS
    configure disables it) and needs C++17.  It is the long term option.
  - Blink SIPManager.py compares settings.audio.sample_rate (an int) with
    the strings '16000' and '32000', so enabling EC always forces 32 kHz.
    Harmless with this patch, but the comparison should use ints.
