Three-Engine Compilation Architecture
Status: Design Phase
Target: v2.1.0
Goal: Position SciTeX Writer as Overleaf competitor through speed, flexibility, and reproducibility
1. Executive Summary
Current State
- Single compilation path: latexmk (currently disabled) with 3-pass fallback
- Bibliography path resolution issues with latexmk +
-output-directory - No speed optimization for iterative editing
- Limited offline capability advantages over Overleaf
Target State
- Three compilation engines available:
- Tectonic: ⚡ Fast mode (1-3s incremental, 10× faster than Overleaf)
- latexmk: 🔧 Standard mode (3-6s, industry standard)
- 3-pass: 🔒 Guaranteed mode (12-18s, maximum compatibility)
- Intelligent auto-detection with graceful degradation
- User-selectable via
--engineflag or config file - Unified error handling and logging
Competitive Advantages
- Speed: Tectonic provides 10× faster compilation for rapid iteration
- Offline: True offline capability (vs Overleaf's cloud-only)
- Reproducibility: Container fallbacks ensure consistent builds
- Choice: Users select speed vs compatibility trade-offs
2. Configuration Schema
2.1 New Config Fields (config_manuscript.yaml)
compilation:
# Engine selection: tectonic, latexmk, 3pass, auto
engine: "auto"
# Auto-detection fallback order
auto_order:
- tectonic # Try fastest first
- latexmk # Fall back to standard
- 3pass # Guaranteed fallback
# Draft mode (single pass, skip bibliography)
draft_mode: false
# Per-engine settings
engines:
tectonic:
# Use incremental compilation cache
incremental: true
# Cache directory (relative to project root)
cache_dir: "./.tectonic-cache"
# Bundle directory for offline packages
bundle_dir: "./.tectonic-bundle"
latexmk:
# Use latexmk's smart recompilation
incremental: true
# Maximum passes before giving up
max_passes: 10
# Fix bibliography path resolution
set_bibinputs: true
3pass:
# Always do full 3 passes (no incremental)
incremental: false
# Verbose output for each pass
verbose_passes: false
# Existing verbosity settings
verbosity:
pdflatex: true
bibtex: true
# Add per-engine verbosity
tectonic: false # Tectonic is verbose by default
latexmk: false
2.2 CLI Flags
# Explicit engine selection
./compile.sh manuscript --engine=tectonic
./compile.sh manuscript --engine=latexmk
./compile.sh manuscript --engine=3pass
# Auto-detection (default)
./compile.sh manuscript --engine=auto
# Engine-specific options
./compile.sh manuscript --engine=tectonic --incremental
./compile.sh manuscript --engine=latexmk --max-passes=15
# Draft mode (works with all engines)
./compile.sh manuscript --draft --engine=tectonic
3. Engine Selection Logic
3.1 Decision Flow
┌─────────────────────────────────────┐
│ User Request: compile manuscript │
└───────────────┬─────────────────────┘
│
▼
┌─────────────────────────────────────┐
│ Check --engine flag │
│ • Explicit: tectonic/latexmk/3pass│
│ • Auto: proceed to detection │
└───────────────┬─────────────────────┘
│
▼
┌──────┴───────┐
│ Explicit? │
└──┬────────┬──┘
Yes No (auto)
│ │
│ ▼
│ ┌─────────────────────┐
│ │ Try engines in │
│ │ auto_order: │
│ │ 1. Check tectonic │
│ │ 2. Check latexmk │
│ │ 3. Use 3pass │
│ └──────┬──────────────┘
│ │
▼ ▼
┌─────────────────────────────────────┐
│ Verify Engine Available │
│ • Command exists? │
│ • Version compatible? │
│ • Container fallback available? │
└───────────────┬─────────────────────┘
│
▼
┌──────┴───────┐
│ Available? │
└──┬────────┬──┘
Yes No
│ │
│ ▼
│ ┌─────────────────────┐
│ │ If explicit: ERROR │
│ │ If auto: try next │
│ └─────────────────────┘
│
▼
┌─────────────────────────────────────┐
│ Execute Compilation │
│ • Set engine-specific env vars │
│ • Run engine-specific commands │
│ • Parse engine-specific errors │
└───────────────┬─────────────────────┘
│
▼
┌──────┴───────┐
│ Success? │
└──┬────────┬──┘
Yes No
│ │
│ ▼
│ ┌─────────────────────┐
│ │ If auto: try next │
│ │ If explicit: report │
│ └─────────────────────┘
│
▼
┌─────────────────────────────────────┐
│ Post-processing │
│ • Validate PDF output │
│ • Report statistics │
│ • Update symlinks │
└─────────────────────────────────────┘
3.2 Implementation (compile_manuscript.sh)
Location: Before line 211 (before PDF generation phase)
# ----------------------------------------
# Engine Selection Phase (NEW)
# ----------------------------------------
echo_header "Engine Selection"
# Get engine from CLI flag or config
SELECTED_ENGINE="${FLAG_ENGINE:-${SCITEX_WRITER_ENGINE}}"
if [ "$SELECTED_ENGINE" = "auto" ]; then
# Auto-detection: try engines in order
source "${SCRIPTS_DIR}/modules/select_compilation_engine.sh"
SELECTED_ENGINE=$(auto_detect_engine)
echo_info "Auto-detected engine: $SELECTED_ENGINE"
else
# Explicit selection: verify availability
source "${SCRIPTS_DIR}/modules/select_compilation_engine.sh"
if ! verify_engine "$SELECTED_ENGINE"; then
echo_error "Requested engine '$SELECTED_ENGINE' not available"
exit 1
fi
echo_info "Using requested engine: $SELECTED_ENGINE"
fi
# Export for downstream modules
export SCITEX_WRITER_SELECTED_ENGINE="$SELECTED_ENGINE"
4. Engine-Specific Implementations
4.1 Command Switching Module Extension
File: scripts/shell/modules/command_switching.src
New functions to add:
# ========================================
# TECTONIC COMMANDS
# ========================================
get_cmd_tectonic() {
local orig_dir="${1:-$(pwd)}"
# Check native tectonic
if command -v tectonic &> /dev/null; then
echo "tectonic"
return 0
fi
# Check container fallback
local container_cmd=$(get_container_cmd "$orig_dir")
if [ -n "$container_cmd" ]; then
echo "$container_cmd tectonic"
return 0
fi
return 1
}
get_tectonic_version() {
local cmd=$(get_cmd_tectonic)
if [ -n "$cmd" ]; then
$cmd --version 2>&1 | head -1
fi
}
# ========================================
# LATEXMK COMMANDS (enhanced)
# ========================================
get_cmd_latexmk() {
local orig_dir="${1:-$(pwd)}"
# Check native latexmk
if command -v latexmk &> /dev/null; then
echo "latexmk"
return 0
fi
# Check module load (HPC systems)
if type module &> /dev/null; then
if module load latexmk 2>&1 | grep -q "latexmk"; then
echo "latexmk"
return 0
fi
fi
# Check container fallback
local container_cmd=$(get_container_cmd "$orig_dir")
if [ -n "$container_cmd" ]; then
echo "$container_cmd latexmk"
return 0
fi
return 1
}
# ========================================
# 3-PASS COMPILATION (existing)
# ========================================
# Already have get_cmd_pdflatex() and get_cmd_bibtex()
# No changes needed
4.2 Engine Selection Module (NEW)
File: scripts/shell/modules/select_compilation_engine.sh
#!/bin/bash
# Select and verify compilation engine
source "$(dirname ${BASH_SOURCE[0]})/command_switching.src"
# Auto-detect best available engine
auto_detect_engine() {
local auto_order="${SCITEX_WRITER_AUTO_ORDER:-tectonic latexmk 3pass}"
for engine in $auto_order; do
if verify_engine "$engine"; then
echo "$engine"
return 0
fi
done
# Fallback to 3pass (always works if pdflatex exists)
echo "3pass"
}
# Verify engine is available
verify_engine() {
local engine="$1"
case "$engine" in
tectonic)
get_cmd_tectonic >/dev/null 2>&1
return $?
;;
latexmk)
# Need both latexmk and pdflatex
get_cmd_latexmk >/dev/null 2>&1 && \
get_cmd_pdflatex >/dev/null 2>&1
return $?
;;
3pass)
# Need pdflatex and bibtex
get_cmd_pdflatex >/dev/null 2>&1 && \
get_cmd_bibtex >/dev/null 2>&1
return $?
;;
*)
echo_error "Unknown engine: $engine"
return 1
;;
esac
}
# Get human-readable engine info
get_engine_info() {
local engine="$1"
case "$engine" in
tectonic)
echo "Tectonic (Fast mode, 1-3s incremental)"
;;
latexmk)
echo "latexmk (Standard mode, 3-6s incremental)"
;;
3pass)
echo "3-pass (Guaranteed mode, 12-18s full)"
;;
esac
}
4.3 Tectonic Engine Implementation (NEW)
File: scripts/shell/modules/engines/compile_tectonic.sh
#!/bin/bash
# Tectonic compilation engine
compile_with_tectonic() {
local tex_file="$1"
local pdf_file="${tex_file%.tex}.pdf"
echo_info "Using Tectonic engine"
# Get tectonic command
local tectonic_cmd=$(get_cmd_tectonic)
if [ -z "$tectonic_cmd" ]; then
echo_error "Tectonic not available"
return 1
fi
# Build tectonic options
local opts=""
# Output directory
local tex_dir=$(dirname "$tex_file")
opts="$opts --outdir=$tex_dir"
# Incremental mode (use cache)
if [ "$SCITEX_WRITER_TECTONIC_INCREMENTAL" = "true" ]; then
opts="$opts --keep-intermediates"
# Set cache directory
if [ -n "$SCITEX_WRITER_TECTONIC_CACHE_DIR" ]; then
export TECTONIC_CACHE_DIR="$SCITEX_WRITER_TECTONIC_CACHE_DIR"
fi
fi
# Bundle directory (offline mode)
if [ -n "$SCITEX_WRITER_TECTONIC_BUNDLE_DIR" ]; then
opts="$opts --bundle=$SCITEX_WRITER_TECTONIC_BUNDLE_DIR"
fi
# Verbosity
if [ "$SCITEX_WRITER_VERBOSE_TECTONIC" != "true" ]; then
opts="$opts --print=error"
fi
# Run compilation
local start=$(date +%s)
echo_info "Running: $tectonic_cmd $opts $tex_file"
local output=$($tectonic_cmd $opts "$tex_file" 2>&1)
local exit_code=$?
local end=$(date +%s)
# Check result
if [ $exit_code -eq 0 ]; then
echo_success "Tectonic compilation: $(($end - $start))s"
return 0
else
echo_error "Tectonic compilation failed (exit code: $exit_code)"
# Show errors if verbose or on failure
if [ "$SCITEX_WRITER_VERBOSE_TECTONIC" = "true" ] || [ $exit_code -ne 0 ]; then
echo "$output"
fi
return 1
fi
}
4.4 latexmk Engine Implementation (FIXED)
File: scripts/shell/modules/engines/compile_latexmk.sh
#!/bin/bash
# latexmk compilation engine with BIBINPUTS fix
compile_with_latexmk() {
local tex_file="$1"
local pdf_file="${tex_file%.tex}.pdf"
echo_info "Using latexmk engine"
# Get latexmk command
local latexmk_cmd=$(get_cmd_latexmk)
if [ -z "$latexmk_cmd" ]; then
echo_error "latexmk not available"
return 1
fi
# Setup paths
local tex_dir=$(dirname "$tex_file")
local project_root=$(git rev-parse --show-toplevel 2>/dev/null || pwd)
# FIX: Set BIBINPUTS to find bibliography files
# latexmk runs bibtex from output directory, need to point to project root
if [ "$SCITEX_WRITER_LATEXMK_SET_BIBINPUTS" = "true" ]; then
export BIBINPUTS="${project_root}:"
echo_info "Set BIBINPUTS=${BIBINPUTS}"
fi
# Build latexmk options
local opts="-pdf -bibtex -interaction=nonstopmode -file-line-error"
# Output directory
opts="$opts -output-directory='$tex_dir'"
# Shell escape for minted, etc.
opts="$opts -pdflatex='pdflatex -shell-escape %O %S'"
# Quiet mode
if [ "$SCITEX_WRITER_VERBOSE_LATEXMK" != "true" ]; then
opts="$opts -quiet"
fi
# Draft mode (single pass)
if [ "$SCITEX_WRITER_DRAFT_MODE" = "true" ]; then
opts="$opts -dvi- -ps-"
echo_info "Draft mode: single pass only"
fi
# Max passes
if [ -n "$SCITEX_WRITER_LATEXMK_MAX_PASSES" ]; then
opts="$opts -latexoption=-interaction=nonstopmode"
fi
# Run compilation
local start=$(date +%s)
local cmd="$latexmk_cmd $opts '$tex_file'"
echo_info "Running: $cmd"
local output=$(eval "$cmd" 2>&1 | grep -v "gocryptfs not found")
local exit_code=$?
local end=$(date +%s)
# Check for critical errors
if echo "$output" | grep -q "Missing bbl file\|failed to resolve\|gave return code"; then
echo_warning "Compilation completed with warnings (check citations/references)"
fi
# Check result
if [ $exit_code -eq 0 ]; then
echo_success "latexmk compilation: $(($end - $start))s"
return 0
else
echo_error "latexmk compilation failed (exit code: $exit_code)"
# Show output if verbose or on failure
if [ "$SCITEX_WRITER_VERBOSE_LATEXMK" = "true" ] || [ $exit_code -ne 0 ]; then
echo "$output"
fi
return 1
fi
}
4.5 3-Pass Engine Implementation (EXISTING)
File: scripts/shell/modules/engines/compile_3pass.sh
Extract from existing compilation_compiled_tex_to_compiled_pdf.sh lines 96-160:
#!/bin/bash
# 3-pass compilation engine (most compatible)
compile_with_3pass() {
local tex_file="$1"
local pdf_file="${tex_file%.tex}.pdf"
echo_info "Using 3-pass engine"
# Get commands
local pdf_cmd=$(get_cmd_pdflatex)
local bib_cmd=$(get_cmd_bibtex)
if [ -z "$pdf_cmd" ] || [ -z "$bib_cmd" ]; then
echo_error "No LaTeX installation found (native, module, or container)"
return 1
fi
# Add compilation options
local tex_dir=$(dirname "$tex_file")
pdf_cmd="$pdf_cmd -output-directory=$tex_dir -shell-escape -interaction=nonstopmode -file-line-error"
# Helper function for timed execution
run_pass() {
local cmd="$1"
local verbose="$2"
local desc="$3"
echo_info "$desc"
local start=$(date +%s)
if [ "$verbose" = "true" ]; then
eval "$cmd" 2>&1 | grep -v "gocryptfs not found"
local ret=${PIPESTATUS[0]}
else
eval "$cmd" >/dev/null 2>&1
local ret=$?
fi
local end=$(date +%s)
echo_info " ($(($end - $start))s)"
return $ret
}
# Main compilation sequence
local total_start=$(date +%s)
local tex_base="${tex_file%.tex}"
local aux_file="${tex_base}.aux"
# Check draft mode
if [ "$SCITEX_WRITER_DRAFT_MODE" = "true" ]; then
# Draft: single pass only
run_pass "$pdf_cmd $tex_file" "$SCITEX_WRITER_VERBOSE_PDFLATEX" "Single pass (draft mode)"
else
# Full: 3-pass compilation
run_pass "$pdf_cmd $tex_file" "$SCITEX_WRITER_VERBOSE_PDFLATEX" "Pass 1/3: Initial"
# Process bibliography if needed
if [ -f "$aux_file" ]; then
if grep -q "\\citation\|\\bibdata\|\\bibstyle" "$aux_file" 2>/dev/null; then
run_pass "$bib_cmd $tex_base" "$SCITEX_WRITER_VERBOSE_BIBTEX" "Processing bibliography"
fi
fi
run_pass "$pdf_cmd $tex_file" "$SCITEX_WRITER_VERBOSE_PDFLATEX" "Pass 2/3: Bibliography"
run_pass "$pdf_cmd $tex_file" "$SCITEX_WRITER_VERBOSE_PDFLATEX" "Pass 3/3: Final"
fi
local total_end=$(date +%s)
echo_success "3-pass compilation: $(($total_end - $total_start))s"
}
4.6 Unified Engine Interface
File: scripts/shell/modules/compilation_compiled_tex_to_compiled_pdf.sh (REFACTORED)
#!/bin/bash
# Main compilation orchestrator - delegates to engine-specific modules
# Source engine implementations
source "$(dirname ${BASH_SOURCE[0]})/engines/compile_tectonic.sh"
source "$(dirname ${BASH_SOURCE[0]})/engines/compile_latexmk.sh"
source "$(dirname ${BASH_SOURCE[0]})/engines/compile_3pass.sh"
compiled_tex_to_pdf() {
echo_info "Converting $SCITEX_WRITER_COMPILED_TEX to PDF..."
local tex_file="$SCITEX_WRITER_COMPILED_TEX"
local engine="$SCITEX_WRITER_SELECTED_ENGINE"
# Dispatch to engine-specific implementation
case "$engine" in
tectonic)
compile_with_tectonic "$tex_file"
;;
latexmk)
compile_with_latexmk "$tex_file"
;;
3pass)
compile_with_3pass "$tex_file"
;;
*)
echo_error "Unknown compilation engine: $engine"
return 1
;;
esac
local ret=$?
# If compilation failed in auto mode, try next engine
if [ $ret -ne 0 ] && [ "$SCITEX_WRITER_AUTO_ENGINE" = "true" ]; then
echo_warning "Engine '$engine' failed, trying fallback..."
# TODO: Implement fallback logic
fi
return $ret
}
# Existing cleanup() function unchanged
cleanup() {
# ... (lines 163-203 unchanged)
}
main() {
compiled_tex_to_pdf
cleanup
}
main
5. Unified Error Handling
5.1 Error Categories
Common errors across all engines:
| Category | Symptoms | Tectonic | latexmk | 3-pass |
|---|---|---|---|---|
| Missing Package | ! LaTeX Error: File not found |
Auto-downloads | Fails | Fails |
| Syntax Error | ! Undefined control sequence |
Shows line | Shows line | Shows line |
| Bibliography | Warning: Citation undefined |
Auto-handles | Needs .bbl | Needs bibtex |
| File Not Found | ! I can't find file |
Clear error | In log | In log |
| Timeout | Process hangs | Rare | Possible | Rare |
5.2 Error Parser
File: scripts/shell/modules/parse_compilation_errors.sh (NEW)
#!/bin/bash
# Parse LaTeX errors from different engines
parse_tectonic_errors() {
local output="$1"
# Tectonic has clean JSON-like error output
echo "$output" | grep "error:" | while read line; do
echo_error "$line"
done
# Extract file:line references
echo "$output" | grep -oP '(?<=error: ).*?:\d+' | head -5
}
parse_latexmk_errors() {
local log_file="$1"
# latexmk errors are in .log file
if [ -f "$log_file" ]; then
# Extract LaTeX errors
grep "^!" "$log_file" | head -5
# Extract missing files
grep "! I can't find file" "$log_file"
# Extract undefined references
grep "Warning.*undefined" "$log_file" | head -3
fi
}
parse_3pass_errors() {
local log_file="$1"
# Same as latexmk (both use pdflatex logs)
parse_latexmk_errors "$log_file"
}
# Unified error reporting
report_compilation_errors() {
local engine="$1"
local tex_file="$2"
local output="$3"
local log_file="${tex_file%.tex}.log"
case "$engine" in
tectonic)
parse_tectonic_errors "$output"
;;
latexmk|3pass)
parse_latexmk_errors "$log_file"
;;
esac
}
5.3 Fallback Logic
When auto-detection is enabled and an engine fails:
- Tectonic fails → Try latexmk
-
Reason: Might be version incompatibility or missing Tectonic-specific features
-
latexmk fails → Try 3-pass
-
Reason: Path resolution issues, latexmk bugs, missing dependencies
-
3-pass fails → Report error and exit
- Reason: Fundamental LaTeX error that no engine can fix
Implementation in compile_with_* functions:
# At end of each engine function, add:
if [ $exit_code -ne 0 ] && [ "$SCITEX_WRITER_AUTO_ENGINE" = "true" ]; then
echo_warning "Engine failed, will try fallback"
return 2 # Special code: try next engine
else
return $exit_code # 0=success, 1=fatal error
fi
6. Testing Strategy
6.1 Test Matrix
| Scenario | Tectonic | latexmk | 3-pass |
|---|---|---|---|
| Simple document (no refs) | ✓ | ✓ | ✓ |
| With bibliography | ✓ | ✓ | ✓ |
| With figures (JPG/PNG) | ✓ | ✓ | ✓ |
| With tables (LaTeX) | ✓ | ✓ | ✓ |
| With minted (shell-escape) | ✓ | ✓ | ✓ |
| Draft mode | ✓ | ✓ | ✓ |
| Syntax error | ✓ | ✓ | ✓ |
| Missing package | ✓ | ✗ | ✗ |
| Missing .bib file | ✗ | ✗ | ✗ |
| Invalid citation | ⚠️ | ⚠️ | ⚠️ |
6.2 Performance Benchmarks
Test document: Current manuscript (~2000 words, 2 figures, 10 citations)
| Metric | Tectonic | latexmk | 3-pass | Target |
|---|---|---|---|---|
| Cold start (first run) | 5-8s | 8-12s | 12-18s | <10s |
| Incremental (text change) | 1-3s | 3-6s | 12-18s | <5s |
| Bibliography update | 2-4s | 4-8s | 12-18s | <8s |
| Figure added | 3-5s | 5-9s | 12-18s | <10s |
| Container fallback | +2-3s | +2-3s | +2-3s | <5s overhead |
6.3 Test Implementation
File: tests/test_compilation_engines.sh (NEW)
#!/bin/bash
# Test all three compilation engines
set -e
SCRIPT_DIR="$(cd $(dirname ${BASH_SOURCE[0]}) && pwd)"
PROJECT_ROOT="$(cd $SCRIPT_DIR/.. && pwd)"
# Test cases
test_simple_document() {
local engine="$1"
echo "Testing simple document with $engine..."
# Create minimal test doc
cat > /tmp/test_simple.tex <<'EOF'
\documentclass{article}
\begin{document}
Hello World
\end{document}
EOF
# Compile with engine
export SCITEX_WRITER_SELECTED_ENGINE="$engine"
$PROJECT_ROOT/scripts/shell/modules/compilation_compiled_tex_to_compiled_pdf.sh /tmp/test_simple.tex
# Check PDF exists
if [ -f /tmp/test_simple.pdf ]; then
echo "✓ Simple document test passed"
return 0
else
echo "✗ Simple document test FAILED"
return 1
fi
}
test_with_bibliography() {
local engine="$1"
echo "Testing bibliography with $engine..."
# Create test doc with citations
cat > /tmp/test_bib.tex <<'EOF'
\documentclass{article}
\begin{document}
Test citation \cite{test2020}.
\bibliography{/tmp/test}
\bibliographystyle{plain}
\end{document}
EOF
cat > /tmp/test.bib <<'EOF'
@article{test2020,
author = {Test Author},
title = {Test Title},
year = {2020}
}
EOF
# Compile
export SCITEX_WRITER_SELECTED_ENGINE="$engine"
$PROJECT_ROOT/scripts/shell/modules/compilation_compiled_tex_to_compiled_pdf.sh /tmp/test_bib.tex
# Check PDF exists and contains reference
if [ -f /tmp/test_bib.pdf ] && pdftotext /tmp/test_bib.pdf - | grep -q "Test Author"; then
echo "✓ Bibliography test passed"
return 0
else
echo "✗ Bibliography test FAILED"
return 1
fi
}
# Run all tests
run_all_tests() {
for engine in tectonic latexmk 3pass; do
echo "========================================"
echo "Testing engine: $engine"
echo "========================================"
test_simple_document "$engine"
test_with_bibliography "$engine"
echo ""
done
}
# Benchmark tests
benchmark_engines() {
echo "Benchmarking compilation speed..."
for engine in tectonic latexmk 3pass; do
echo "Engine: $engine"
# Cold start
rm -f /tmp/test_simple.pdf /tmp/test_simple.aux
start=$(date +%s.%N)
test_simple_document "$engine" >/dev/null 2>&1
end=$(date +%s.%N)
cold=$(echo "$end - $start" | bc)
echo " Cold start: ${cold}s"
# Incremental (modify tex, recompile)
echo "Modified" >> /tmp/test_simple.tex
start=$(date +%s.%N)
test_simple_document "$engine" >/dev/null 2>&1
end=$(date +%s.%N)
incremental=$(echo "$end - $start" | bc)
echo " Incremental: ${incremental}s"
echo ""
done
}
# Main
case "${1:-all}" in
all)
run_all_tests
;;
benchmark)
benchmark_engines
;;
*)
echo "Usage: $0 [all|benchmark]"
exit 1
;;
esac
7. Migration Path
Phase 1: Add Tectonic (v2.1.0)
Status: NEW feature, opt-in
# Default remains 3-pass (safe, compatible)
compilation:
engine: "3pass"
# Users can opt-in to Tectonic
compilation:
engine: "tectonic"
Deliverables:
- [ ] Tectonic command detection
- [ ] compile_tectonic.sh implementation
- [ ] Configuration options
- [ ] Documentation
- [ ] Basic tests
Risk: Low (new feature, doesn't break existing workflows)
Phase 2: Fix latexmk (v2.1.1)
Status: BUG FIX, automatic
# Enable latexmk with BIBINPUTS fix
compilation:
engine: "latexmk"
Deliverables:
- [ ] BIBINPUTS environment variable fix
- [ ] compile_latexmk.sh implementation
- [ ] Test with bibliography path resolution
- [ ] Re-enable in compilation_compiled_tex_to_compiled_pdf.sh
Risk: Medium (modifies existing behavior, but fixes known bug)
Phase 3: Auto-detection (v2.2.0)
Status: BREAKING CHANGE (default behavior changes)
# New default: auto-detect best engine
compilation:
engine: "auto"
auto_order:
- tectonic # Fastest
- latexmk # Standard
- 3pass # Guaranteed
Deliverables:
- [ ] Auto-detection logic
- [ ] Fallback chain implementation
- [ ] Migration guide for users who want old behavior
- [ ] Comprehensive testing
Risk: High (changes default compilation, needs thorough testing)
Phase 4: Optimization (v2.3.0)
Status: ENHANCEMENTS
Features:
- [ ] Parallel compilation for multiple documents
- [ ] Compilation cache management
- [ ] Pre-warming Tectonic bundle for offline use
- [ ] Benchmark dashboard
- [ ] Engine recommendation based on document complexity
Risk: Low (optional optimizations)
8. Documentation Requirements
8.1 User-Facing Documentation
File: docs/compilation-engines.md (NEW)
Topics to cover:
1. Overview of three engines
2. When to use each engine
3. How to select engine (CLI flag, config file)
4. Troubleshooting engine-specific issues
5. Performance comparison
6. Offline usage (Tectonic bundle)
8.2 Developer Documentation
File: docs/dev/compilation-architecture.md (NEW)
Topics to cover:
1. Engine interface specification
2. Adding new engines
3. Error handling patterns
4. Testing new engines
5. Container integration
8.3 Migration Guide
File: docs/migration/v2.0-to-v2.1.md (NEW)
Topics to cover:
1. What changed in v2.1
2. How to opt-in to new engines
3. Rollback procedure
4. Known issues
5. Performance improvements
9. Implementation Checklist
Infrastructure (Week 1)
- [ ] Create engine-specific modules directory:
scripts/shell/modules/engines/ - [ ] Create
select_compilation_engine.shmodule - [ ] Extend
command_switching.srcwith Tectonic/latexmk detection - [ ] Add configuration schema to
config_manuscript.yaml - [ ] Add CLI flag parsing for
--engine
Tectonic Engine (Week 1-2)
- [ ] Implement
compile_tectonic.sh - [ ] Test with simple document
- [ ] Test with bibliography
- [ ] Test with figures
- [ ] Container fallback for Tectonic
latexmk Engine (Week 2)
- [ ] Implement
compile_latexmk.shwith BIBINPUTS fix - [ ] Test bibliography path resolution
- [ ] Test incremental compilation
- [ ] Verify all flags work (--draft, --verbose, etc.)
3-Pass Engine (Week 2)
- [ ] Extract from existing code to
compile_3pass.sh - [ ] No functional changes (just refactoring)
- [ ] Ensure backward compatibility
Integration (Week 3)
- [ ] Refactor
compilation_compiled_tex_to_compiled_pdf.shto dispatch - [ ] Implement auto-detection logic
- [ ] Implement fallback chain
- [ ] Unified error reporting
Testing (Week 3-4)
- [ ] Create
test_compilation_engines.sh - [ ] Test all engines with all scenarios
- [ ] Performance benchmarks
- [ ] Container tests (Singularity/Docker)
- [ ] Integration tests with full manuscript compilation
Documentation (Week 4)
- [ ] User guide:
docs/compilation-engines.md - [ ] Developer guide:
docs/dev/compilation-architecture.md - [ ] Migration guide:
docs/migration/v2.0-to-v2.1.md - [ ] Update README with engine selection examples
- [ ] Update CHANGELOG
Release (Week 4)
- [ ] Version bump to v2.1.0
- [ ] Release notes highlighting three engines
- [ ] Performance comparison vs Overleaf
- [ ] Social media announcement (if applicable)
10. Success Criteria
Functional Requirements
✅ All three engines compile successfully
✅ Auto-detection works reliably
✅ Fallback chain handles failures gracefully
✅ Configuration overrides work (CLI flag > config file > default)
✅ All existing features work with all engines (draft mode, diff, etc.)
✅ Container fallbacks work for all engines
Performance Requirements
✅ Tectonic: <3s incremental compilation
✅ latexmk: <6s incremental compilation
✅ 3-pass: <18s full compilation
✅ Auto-detection overhead: <500ms
Quality Requirements
✅ Zero regressions in existing tests
✅ 100% test coverage for new engine modules
✅ Documentation complete and accurate
✅ No breaking changes for existing users (until v2.2.0 auto-detect default)
Competitive Requirements
✅ Tectonic 10× faster than Overleaf for incremental edits
✅ True offline capability (Tectonic bundle)
✅ Reproducible builds (container fallbacks)
✅ User choice (flexibility not available in Overleaf)
11. Risk Assessment
| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| Tectonic version incompatibility | Medium | High | Container fallback, version pinning |
| latexmk BIBINPUTS doesn't fix all path issues | Medium | Medium | Keep 3-pass fallback, thorough testing |
| Users confused by three engines | Low | Medium | Clear documentation, sensible defaults |
| Performance regression | Low | High | Benchmarks before/after, A/B testing |
| Container overhead too high | Low | Medium | Optimize container warmup, cache layers |
| Breaking changes in Phase 3 (auto-detect) | Medium | High | Migration guide, deprecation warnings |
12. Open Questions
- Tectonic bundle management: How do we pre-warm the bundle for offline use?
- Option A: Include pre-built bundle in repository (large)
- Option B: Download on first run (requires internet)
- Option C: Provide
make tectonic-bundlecommand -
Decision: Option C, document in offline usage guide
-
Container priority: Should Tectonic-in-container be tried before latexmk-native?
- Current: native > module > container (per command)
- Proposed: Respect engine priority even across container boundary
-
Decision: TBD, test both approaches
-
Error threshold for fallback: How many errors before trying next engine?
- Option A: Any non-zero exit code
- Option B: Only specific error types (missing package, timeout)
-
Decision: Option A for auto-detect, Option B for explicit engine
-
Parallel compilation: Can we run multiple engines in parallel and pick fastest?
- Pros: Maximum speed (first to finish wins)
- Cons: Resource intensive, confusing logs
- Decision: Defer to v2.3.0 optimization phase
13. Next Steps
Immediate (this week):
1. Create infrastructure (engine modules, config schema)
2. Implement Tectonic engine
3. Basic testing with simple documents
Short-term (next 2 weeks):
1. Implement latexmk with BIBINPUTS fix
2. Refactor 3-pass to separate module
3. Integration testing
Medium-term (next month):
1. Auto-detection and fallback logic
2. Comprehensive testing
3. Documentation
4. v2.1.0 release
Long-term (3-6 months):
1. Performance optimizations
2. Offline bundle management
3. Benchmark vs Overleaf
4. Marketing materials
Appendix A: File Structure Changes
scitex-writer/
├── config/
│ └── config_manuscript.yaml # Add engine selection
├── scripts/
│ └── shell/
│ ├── compile_manuscript.sh # Add engine selection phase
│ └── modules/
│ ├── command_switching.src # Add Tectonic/latexmk detection
│ ├── select_compilation_engine.sh # NEW: Engine selection
│ ├── parse_compilation_errors.sh # NEW: Error parsing
│ ├── compilation_compiled_tex_to_compiled_pdf.sh # REFACTOR: Dispatch only
│ └── engines/ # NEW: Engine implementations
│ ├── compile_tectonic.sh
│ ├── compile_latexmk.sh
│ └── compile_3pass.sh
├── tests/
│ └── test_compilation_engines.sh # NEW: Engine tests
└── docs/
├── compilation-engines.md # NEW: User guide
├── dev/
│ └── compilation-architecture.md # NEW: Developer guide
└── migration/
└── v2.0-to-v2.1.md # NEW: Migration guide
Appendix B: Configuration Example
# config_manuscript.yaml (v2.1.0+)
# Document metadata
document:
type: "manuscript"
version: "v2.1.0"
# Compilation engine selection
compilation:
# Engine: tectonic, latexmk, 3pass, auto
engine: "auto"
# Auto-detection fallback order
auto_order:
- tectonic
- latexmk
- 3pass
# Draft mode (single pass, skip bibliography)
draft_mode: false
# Per-engine settings
engines:
tectonic:
incremental: true
cache_dir: "./.tectonic-cache"
bundle_dir: "./.tectonic-bundle"
latexmk:
incremental: true
max_passes: 10
set_bibinputs: true
3pass:
incremental: false
verbose_passes: false
# Verbosity settings
verbosity:
pdflatex: true
bibtex: true
tectonic: false # Tectonic is verbose by default
latexmk: false
# Citation style
citation:
style: "unsrtnat"
# Paths
paths:
manuscript_dir: "./01_manuscript"
compiled_tex: "./01_manuscript/manuscript.tex"
compiled_pdf: "./01_manuscript/manuscript.pdf"
doc_log_dir: "./01_manuscript/logs"
versions_dir: "./01_manuscript/archive"
root_dir: "."
Appendix C: CLI Examples
# Auto-detect best engine (default in v2.2.0+)
./compile.sh manuscript
# Explicit engine selection
./compile.sh manuscript --engine=tectonic
./compile.sh manuscript --engine=latexmk
./compile.sh manuscript --engine=3pass
# Draft mode with Tectonic (fastest iteration)
./compile.sh manuscript --draft --engine=tectonic
# Verbose output for debugging
./compile.sh manuscript --engine=latexmk --verbose
# Force container execution
./compile.sh manuscript --engine=tectonic --container
# Check available engines
./compile.sh manuscript --list-engines
# Output:
# ✓ tectonic (v0.14.1) [Fast mode: 1-3s]
# ✓ latexmk (v4.77) [Standard mode: 3-6s]
# ✓ 3pass (native) [Guaranteed mode: 12-18s]
# Benchmark all engines
./compile.sh manuscript --benchmark
# Output:
# tectonic: 2.3s (incremental: 1.1s)
# latexmk: 4.7s (incremental: 3.2s)
# 3pass: 14.2s (incremental: 14.2s)
END OF ARCHITECTURE DESIGN
This document will be updated as implementation progresses.