docs: add analytics and tts design spec

2026-05-17 09:54:17 +08:00
parent eb83384623
commit 73c292b7c2
1 changed files with 403 additions and 0 deletions
@@ -0,0 +1,403 @@
+# Analytics and Text-to-Speech Design
+
+Date: 2026-05-17
+
+## Goal
+
+Add two complete but independently shippable capabilities:
+
+1. Mini program analytics: collect user behavior events from the user-facing mini program and show behavior, funnel, and preference statistics in `web-admin`.
+2. Script text-to-speech: let users listen to generated scripts and creative content through a CPU-friendly open-source TTS service deployed on `101.200.208.45`.
+
+The implementation should fit the current project shape:
+
+- `backend-single`: Spring Boot 2.7, MyBatis Plus, MySQL, Redis.
+- `mini-program`: uni-app/Vue.
+- `web-admin`: Vue, Element Plus, ECharts.
+- Deployment target: `101.200.208.45` with passwordless SSH.
+
+## Scope
+
+### Change 1: Analytics
+
+The mini program gets a lightweight analytics SDK. It collects automatic page/session events and explicit business events, then sends batched events to the backend. The backend stores raw events in MySQL and exposes admin-only aggregation APIs. `web-admin` adds a behavior analytics page.
+
+### Change 2: Text-to-Speech
+
+The mini program gets a script reading control on the script detail page. The backend owns task creation, permission checks, text cleaning, audio cache lookup, and public audio URLs. A local Python FastAPI service backed by an open-source CPU-friendly TTS engine generates the audio files.
+
+TTS-related actions are also tracked as analytics events, so admins can see which content is most often requested and played.
+
+## Recommended Approach
+
+Use the lightweight in-project approach rather than adding a separate analytics platform or data warehouse.
+
+Reasons:
+
+- The requested analysis is business-specific: script style, inspiration prompt, life event tags, path mapping, and reading behavior.
+- MySQL aggregation is enough for the first version and matches the existing backend.
+- Operational complexity stays low.
+- The TTS service can be swapped later without changing mini program behavior.
+
+For TTS, use MeloTTS or another CPU-friendly open-source Chinese TTS service for the first version. The server has no GPU, so heavier real-time models such as CosyVoice are not the default production path. The backend interface should remain engine-neutral so a better engine can replace it later.
+
+## Data Model
+
+### `t_analytics_event`
+
+Stores raw user behavior events.
+
+Fields:
+
+- `id`: primary key.
+- `user_id`: nullable current user id from token.
+- `anonymous_id`: long-lived local mini program id for anonymous or pre-login behavior.
+- `session_id`: id for one app launch/session.
+- `event_name`: event key, for example `page_view`, `script_generate_success`.
+- `event_type`: broad category, for example `page`, `script`, `life_event`, `tts`.
+- `page_path`: current page.
+- `referrer_path`: previous page.
+- `properties`: JSON object for business attributes such as script style, length, tags, button source, error code.
+- `device_info`: JSON object for platform, model, system version, SDK version, WeChat version.
+- `duration_ms`: optional page stay time, generation time, or playback time.
+- `occurred_at`: client event time.
+- `server_time`: backend receive time.
+- common fields: `create_time`, `update_time`, `is_deleted`, `remarks`.
+
+Indexes:
+
+- `idx_analytics_event_name`
+- `idx_analytics_event_type`
+- `idx_analytics_event_user_id`
+- `idx_analytics_event_anonymous_id`
+- `idx_analytics_event_occurred_at`
+- `idx_analytics_event_name_time (event_name, occurred_at)`
+- `idx_analytics_event_user_time (user_id, occurred_at)`
+
+### `t_tts_task`
+
+Stores text-to-speech generation tasks and cache metadata.
+
+Fields:
+
+- `id`: primary key.
+- `user_id`: owner.
+- `source_type`: first version uses `epic_script`.
+- `source_id`: related script id.
+- `text_hash`: hash of cleaned text plus voice.
+- `text_length`: cleaned text length.
+- `voice`: selected voice, default Mandarin voice.
+- `status`: `pending`, `processing`, `success`, `failed`.
+- `audio_url`: URL returned to mini program.
+- `audio_path`: server file path.
+- `duration_ms`: generated audio duration if available.
+- `error_message`: failure reason for retry/debug.
+- `request_count`: number of times this cached audio has been requested.
+- common fields: `create_time`, `update_time`, `is_deleted`, `remarks`.
+
+Indexes:
+
+- `idx_tts_task_user_source (user_id, source_type, source_id)`
+- `idx_tts_task_text_hash (text_hash)`
+- `idx_tts_task_status (status)`
+- `idx_tts_task_create_time (create_time)`
+
+Successful tasks with the same `text_hash + voice` should reuse audio instead of regenerating.
+
+## Backend API Design
+
+### Mini Program Analytics API
+
+`POST /analytics/events/batch`
+
+Accepts batched mini program events. It is available before and after login. If a token is present, the backend fills `user_id`; otherwise the event is stored with `anonymous_id`.
+
+Request shape:
+
+```json
+{
+  "anonymousId": "anon_xxx",
+  "sessionId": "session_xxx",
+  "deviceInfo": {},
+  "events": [
+    {
+      "eventName": "script_detail_view",
+      "eventType": "script",
+      "pagePath": "/pages/main/ScriptDetailView",
+      "referrerPath": "/pages/main/ScriptView",
+      "properties": {},
+      "durationMs": 12000,
+      "occurredAt": "2026-05-17 09:30:00"
+    }
+  ]
+}
+```
+
+Validation rules:
+
+- Max 50 events per request.
+- Event names must match a safe naming pattern such as lowercase letters, digits, and underscores.
+- `properties` and `device_info` have strict size limits.
+- Sensitive fields are not accepted.
+- Invalid individual events can be skipped while valid events are stored, with a response count.
+
+### Admin Analytics API
+
+All endpoints are under `/admin/analytics/**` and require admin auth.
+
+- `GET /admin/analytics/overview`: PV, UV, event count, active users, average stay time, TTS request/play counts.
+- `GET /admin/analytics/trend`: daily or hourly event trend for selected event names.
+- `GET /admin/analytics/funnel`: key funnel steps, first version:
+  `app_launch -> profile_complete -> life_event_create -> script_generate_success -> path_select`.
+- `GET /admin/analytics/preferences`: script style, script length, inspiration prompt type, life event tags, and TTS source popularity.
+- `GET /admin/analytics/top-events`: hot events and error events.
+- `GET /admin/analytics/users`: user behavior ranking by event count, recent activity, and selected event filters.
+
+Default query window is 7 days. Maximum query window is 90 days.
+
+### Mini Program TTS API
+
+`POST /tts/tasks`
+
+Creates or reuses a TTS task.
+
+Request:
+
+```json
+{
+  "sourceType": "epic_script",
+  "sourceId": "script_id",
+  "voice": "default_zh_female"
+}
+```
+
+Behavior:
+
+- Validate the current user owns the source content.
+- Load the script content from `t_epic_script`.
+- Clean Markdown and unsupported symbols.
+- Enforce text length limit.
+- Compute `text_hash`.
+- If a successful matching task exists, return `success` and `audioUrl`.
+- If a pending or processing matching task exists, return its task id.
+- Otherwise create a pending task and process it asynchronously.
+
+`GET /tts/tasks/{id}`
+
+Returns task status and audio URL if successful.
+
+`GET /tts/tasks/by-source`
+
+Checks whether a source already has a TTS task for the current user and selected voice.
+
+### Internal Python TTS API
+
+The Python service listens only on localhost or private network.
+
+`POST http://127.0.0.1:19110/synthesize`
+
+Request:
+
+```json
+{
+  "text": "cleaned text",
+  "voice": "default_zh_female",
+  "outputPath": "/data/uploads/emotion-museum/tts/hash.mp3"
+}
+```
+
+Response:
+
+```json
+{
+  "success": true,
+  "audioPath": "/data/uploads/emotion-museum/tts/hash.mp3",
+  "durationMs": 123456,
+  "engine": "melotts"
+}
+```
+
+The Python service is not exposed directly to the mini program.
+
+## Mini Program Design
+
+### Analytics SDK
+
+Add `mini-program/src/services/analytics.js`.
+
+Responsibilities:
+
+- `initAnalytics()`: create or restore `anonymous_id`, create `session_id`, capture device info.
+- `track(eventName, properties)`: queue a business event.
+- `trackPageView(pagePath, options)`: record page enter.
+- `trackPageLeave(pagePath)`: record page leave and stay duration.
+- `flush()`: send queued events to `/analytics/events/batch`.
+
+Queue behavior:
+
+- Keep at most 100 local queued events.
+- Flush every 10 seconds, when 20 events are queued, when the app goes to background, and before page unload where supported.
+- Failed requests keep the queue for retry.
+- Analytics failures never block product flows.
+
+Initial event coverage:
+
+- App/session: launch, show, hide.
+- Page: view and leave for main pages.
+- Script: inspiration view/click, generate start/success/failure, detail view, select/path mapping.
+- Life event: create, update, AI assist, favorite, share.
+- TTS: request, task success, play, pause, complete, error.
+
+### TTS Player UI
+
+Add a script-specific audio component, for example `ScriptAudioPlayer.vue`, instead of mixing with the existing background `MusicPlayer.vue`.
+
+States:
+
+- No audio: show generate reading action.
+- Pending/processing: show generating state.
+- Success: show play/pause.
+- Failed: show retry.
+
+Playback uses `uni.createInnerAudioContext()`. The component releases audio resources when leaving the page.
+
+`ScriptDetailView.vue` adds this component in the hero area or bottom action area.
+
+## Web Admin Design
+
+Add route and menu item for `/analytics`.
+
+Create a first-version behavior analytics page with existing Element Plus and ECharts patterns.
+
+Sections:
+
+- Overview cards: PV, UV, active users, event count, average stay, TTS requests, TTS plays.
+- Trend chart: selected core events by day.
+- Funnel chart: launch to profile, event creation, script generation, path selection.
+- Preference panels: script style, script length, inspiration type, life event tags, most-read content.
+- Top events table: event name, count, users, failure rate where applicable.
+
+The page must handle empty data, sparse data, and loading/error states.
+
+## Privacy and Safety
+
+Do not collect:
+
+- Phone numbers.
+- Tokens.
+- SMS codes.
+- Full script text.
+- Full life event text.
+- Long exception stack traces.
+- Any sensitive identity fields.
+
+Preference analysis stores only enumerations and small metadata, for example `style=career`, `length=medium`, `tag=growth`, `word_count=1200`.
+
+First version user analysis is behavior aggregation only, not sensitive psychological or identity inference.
+
+## Performance and Reliability
+
+Analytics:
+
+- Batch insert events.
+- Keep admin queries bounded by date range.
+- Default analytics range is 7 days.
+- Maximum analytics range is 90 days.
+- Add indexes for event/time and user/time queries.
+- Consider daily aggregate tables only after raw event volume becomes a problem.
+
+TTS:
+
+- Per-user generation rate limit, for example 2 per minute and 20 per day.
+- Text length limit, for example 5000 cleaned characters.
+- Python TTS concurrency limit of 1 or 2 workers on CPU.
+- Task timeout marks the task as `failed`.
+- Audio cache uses `text_hash + voice`.
+- Old unused audio files can be cleaned later by scheduled job.
+- Python service is private and managed with systemd.
+
+## Deployment
+
+Change 1 deployment:
+
+1. Add SQL migration for analytics table.
+2. Deploy backend analytics write API.
+3. Deploy admin analytics aggregation API.
+4. Deploy web-admin analytics page.
+5. Deploy mini program analytics SDK and event coverage.
+
+Change 2 deployment:
+
+1. Install Python runtime and TTS service on `101.200.208.45`.
+2. Add systemd service for the TTS FastAPI app.
+3. Add health check for local TTS service.
+4. Add SQL migration for TTS task table.
+5. Deploy backend TTS task APIs.
+6. Deploy mini program audio player.
+7. Connect TTS events to analytics.
+
+## Testing Plan
+
+Analytics backend:
+
+- Batch event insert succeeds.
+- Anonymous events are accepted.
+- Logged-in events fill `user_id`.
+- Oversized batches are rejected or capped.
+- Oversized JSON properties are rejected.
+- Invalid event names are rejected.
+- Admin overview/trend/funnel/preference calculations are correct on seeded data.
+
+Mini program analytics:
+
+- Launch and background events flush.
+- Page view and leave events include duration.
+- Script generation and detail events are captured.
+- Network failure keeps queued events and retries later.
+- Analytics failures do not show user-facing errors.
+
+Web admin:
+
+- Empty data renders cleanly.
+- Sparse data renders cleanly.
+- Seven-day trends render.
+- Filters refresh data.
+- Permission guard works through existing admin auth.
+
+TTS backend:
+
+- Current user can create task for own script.
+- User cannot create task for another user's script.
+- Existing successful cache returns immediately.
+- Pending task returns task id.
+- Failed task exposes retry path.
+- Text length limit works.
+- Python service failure marks task failed.
+
+Python TTS:
+
+- Short Chinese text generates audio.
+- Long Chinese text within limit generates audio.
+- Output file exists at requested path.
+- Health check returns healthy.
+- CPU and memory usage are acceptable on the server.
+
+Mini program TTS:
+
+- Detail page shows correct state for no task, generating, success, and failed.
+- Play, pause, complete, and error events are tracked.
+- Leaving page releases audio resources.
+
+## Open Decisions for Implementation
+
+- Exact TTS engine package and model version after a quick server resource check.
+- Exact voice identifiers exposed in the first version.
+- Whether long scripts should be truncated in version one or split into multiple audio segments.
+- Whether admin analytics should live as one `/analytics` page or split into sub-tabs.
+
+The default implementation should choose the simplest working options:
+
+- MeloTTS/FastAPI for CPU-friendly generation.
+- One default Mandarin voice.
+- Text length limit with clear retry/failure message, no segment stitching in version one.
+- One admin analytics page with internal sections.