Why I Built This
When I started planning my undergraduate thesis at Telkom University, I wanted to tackle misinformation on social media, focusing on Indonesian economic discourse. Topics like IHSG index movements, Rupiah depreciation, and the launch of the sovereign wealth fund Danantara Indonesia frequently triggered polarized commentary and misleading claims.
Most credibility detection studies I read looked only at tweet text. In practice, false claims on platforms like X (formerly Twitter) rarely rely on text alone. People share misleading screenshots, edited charts, and memes alongside their captions. At the same time, account history (follower count, verification status, and posting behavior) provides subtle cues about authenticity that text alone cannot capture.
The core question behind my thesis was: can fusing text, image features, and user metadata into a single deep learning pipeline improve credibility classification accuracy over standard text-only baselines?
I later refined this work into a paper and published it at IEEE ICICyTA 2025.
How I Built the Pipeline
1. Data Collection and Labelling
I collected 23,564 tweets from X between January 2024 and September 2025 using the Tweet-Harvest library, targeting keywords like “IHSG,” “Pelemahan Rupiah,” “Investasi,” “#EkonomiIndonesia,” and “#Danantara.” Each tweet came bundled with its text, attached image, and user metadata, making it a naturally multimodal dataset.
Three annotators labeled each tweet as credible or non-credible using majority voting to reduce bias, following Castillo et al.’s definition of credible content as factual, informative, and sourced from verified accounts. The final dataset was evenly balanced:
| Label | Count | Percentage |
|---|---|---|
| Credible | 11,782 | 50% |
| Non-credible | 11,782 | 50% |
2. Text Representation: TF-IDF + Custom GloVe
Because standard pre-trained embeddings often struggle with Indonesian social media slang and finance-specific jargon, I trained a custom GloVe embedding model directly on domain data instead of relying on off-the-shelf vectors.
I built a domain-specific corpus of 62,274 records by combining the 23,564 collected tweets with 38,710 articles scraped from IndoNews, pairing informal social vocabulary with formal news reporting:
| Corpus Source | Total Records |
|---|---|
| Tweets | 23,564 |
| IndoNews Articles | 38,710 |
| Combined Corpus | 62,274 |
These embeddings expanded the TF-IDF feature matrix by substituting zero-weight terms with semantically similar words identified by GloVe to reduce sparsity. For example, the top semantic neighbors GloVe found for “Danantara” were investasi, ekonomi, indonesia, kelola, and hadir.
3. Multimodal Fusion Architecture
I combined three feature streams into a single classification network:
- Text stream: TF-IDF for lexical frequency fused with GloVe for semantic expansion
- Visual stream: 1024-dimensional image feature maps extracted using MobileNetV1
- Metadata stream: follower count, verification status, account age, and engagement metrics (quotes, replies, retweets, and favorites)
These fused features were fed into a hybrid CNN + BiGRU network, tested in two configurations:
- CNN + BiGRU: convolution and pooling first to extract local patterns, then BiGRU for broader contextual modeling
- BiGRU + CNN: BiGRU first for contextual dependencies, then Conv1D and Global Max Pooling to condense local patterns
4. Automated Tuning with Particle Swarm Optimization
Instead of manually tuning hyperparameters through trial and error, I implemented Particle Swarm Optimization (PSO). Each particle in the swarm represents a candidate hyperparameter configuration, updating its position based on its own best-known result and the swarm’s global best.
The search space covered six parameters:
| Hyperparameter | Range |
|---|---|
| GRU units | 32 to 256 |
| Filters | 32 to 256 |
| Kernel size | 2 to 5 |
| Dropout | 0.2 to 0.6 |
| Dense units | 32 to 128 |
| Learning rate | 1e-5 to 1e-3 |
Experimental Journey
The thesis ran through five sequential experiment scenarios, each narrowing down the best configuration for the next stage:
| Scenario | Goal | Key Result |
|---|---|---|
| 1. Data split ratio | Find best train/test split | 90:10 split won |
| 2. TF-IDF feature size | Find optimal feature dimensionality | CNN: 3,000 features / BiGRU: 1,000 features |
| 3. GloVe expansion + hybrid models | Reduce TF-IDF sparsity, compare CNN-BiGRU vs BiGRU-CNN | BiGRU-CNN: 75.92% / CNN-BiGRU: 75.78% |
| 4. Multimodal fusion | Add image and user-based features | CNN-BiGRU jumped to 78.43% |
| 5. PSO hyperparameter tuning | Fine-tune the best multimodal model | CNN-BiGRU reached 79.09% |
Findings and Results
| Model Configuration | Accuracy |
|---|---|
| CNN-BiGRU + PSO (proposed) | 79.09% |
| BiGRU-CNN + PSO | 77.93% |
| CNN-BiGRU baseline (text only, TF-IDF) | 73.48% |
Integrating visual and account metadata produced a +4.95% accuracy gain over the text-only TF-IDF baseline (73.48% to 78.43%, before PSO tuning).
Looking at the best model’s confusion behavior:
| Model | Label | Precision | Recall | F1-Score |
|---|---|---|---|---|
| CNN-BiGRU | Non-credible (0) | 78.14% | 80.97% | 79.45% |
| CNN-BiGRU | Credible (1) | 80.38% | 77.22% | 78.68% |
| BiGRU-CNN | Non-credible (0) | 79.22% | 75.76% | 77.43% |
| BiGRU-CNN | Credible (1) | 76.79% | 80.10% | 78.40% |
CNN + BiGRU turned out to be the more cautious model. It caught non-credible content reliably (80.97% recall) and its “credible” predictions were solid (80.38% precision). In credibility detection, keeping false positives low is essential: mistakenly labeling misleading content as trustworthy does more harm than flagging an ambiguous post for review.
Limitations and Future Work
- The dataset only covers Indonesian text, without regional dialects or multilingual test cases.
- The multimodal component is limited to static images; video and audio were not explored.
- TF-IDF remains non-contextual compared to modern transformer-based embeddings like IndoBERT.
Next steps include testing transformer-based embeddings, expanding the domain vocabulary corpus, improving annotation quality through crowdsourcing, and building a real-time credibility detection system for practical deployment.
Publication
This research was accepted and published in the peer-reviewed proceedings of the IEEE International Conference on ICT for Smart Society (ICICyTA) 2025:
- Title: Multimodal Information Credibility on Social Media X Using a Hybrid CNN-BiGRU with Particle Swarm Optimization
- DOI: 10.1109/ICICyTA68677.2025.11362759
- Authors: Gede Bagus Krishnanditya Merta, Erwin Budi Setiawan
- Institution: Telkom University, School of Computing
- Academic Honors: Summa Cum Laude (GPA 3.96/4.00)
- Tech stack: Python, PyTorch, TensorFlow, Scikit-learn, CNN, BiGRU, GloVe, TF-IDF, MobileNetV1, Particle Swarm Optimization