Automating News and Keyword Collection for AI Blogging

![[Node.JS 자동화] Node.js 자동화 프로그래밍, 반복 작업을 AI와 함께 자동화하는 실전 개발(프롤로그)](/uploads/editor/Node-JS-%EC%9E%90%EB%8F%99%ED%99%94-Node-js-%EC%9E%90%EB%8F%99%ED%99%94-%ED%94%84%EB%A1%9C%EA%B7%B8%EB%9E%98%EB%B0%8D-%EB%B0%98%EB%B3%B5-%EC%9E%91%EC%97%85%EC%9D%84-AI%EC%99%80-%ED%95%A8%EA%BB%98-%EC%9E%90%EB%8F%99%ED%99%94%ED%95%98%EB%8A%94-%EC%8B%A4%EC%A0%84-%EA%B0%9C%EB%B0%9C-%ED%94%84%EB%A1%A4%EB%A1%9C%EA%B7%B8-20260702-e78eab.webp)
Automated Posting Starts with the Right Keywords
No matter how good an AI writes, the choice of keywords significantly impacts results. Keywords with no search volume won't attract visitors, and overused or outdated topics struggle to gain attention.
Thus, an automated posting system must begin by finding quality keywords.
In this lesson, we'll automate the entire process.
Where to Collect Keywords From?
Keywords can be gathered from various sources:
News websites
Popular search terms on portals
Internal databases
RSS feeds
Open APIs
Social media trends
Using multiple sources tailored to your project's purpose is most effective.
Overall Workflow
This project operates in the following sequence:
Keyword collection
↓
Deduplication check
↓
Remove irrelevant keywords
↓
Category classification
↓
Data storage in database
↓
Wait for AI to generate posts
This cycle repeats at regular intervals.
Collecting Data via APIs with Axios
For websites offering APIs, using Axios is the fastest and most stable approach.
For example:
const axios = require("axios");
const response = await axios.get(apiUrl);
console.log(response.data);API-based methods are fast and easy to maintain.
Data Collection Using Playwright
For sites without APIs, Playwright can automate browser actions like:
Searching
Clicking
Scrolling
Extracting data
This is especially useful for JavaScript-heavy sites.
Collect Only Necessary Data
You don't need to store all information. For automated posting, only the essential details are required:
Title
URL
Date published
Keywords
Category
Storing minimal data improves database efficiency and processing speed.
Keyword Cleanup Process
Collected keywords must be cleaned before use:
Remove leading/trailing spaces
Eliminate special characters
Deduplicate entries
Exclude overly short phrases
Remove meaningless terms
This ensures the AI generates better content.
Removing Duplicate Keywords
Repeated keywords should be avoided in automated posting. For instance, exclude keywords used within the last 30 days to prevent duplicate topics from being generated repeatedly.
This is a common practice in real-world applications.
Automatic Category Classification
Collected keywords should also be categorized. For example:
Technology
Economy
Health
Travel
Sports
Current Affairs
Automatic categorization helps AI generate more natural content. Many developers use OpenAI API for automatic category classification.
Data Storage in Database
After cleaning, keywords are stored in a database with additional metadata:
Keyword
Category
Date collected
Usage status
View count
Status
This structured data allows AI to generate posts efficiently.
Connecting with a Scheduler
The keyword collection program can be scheduled using Cron to run automatically:
Every 10 minutes
Every 30 minutes
Every hour
This eliminates the need for manual execution.
Maintaining Logs
Logging is essential for automated programs. For example:
[10:00]
News collection started
[10:01]
42 keywords collected
[10:02]
Deduplication complete
[10:03]
DB savedThis makes troubleshooting easier when issues arise.
Telegram Notifications
Results can be sent via Telegram after collection. For example:
News collection completed
New keywords: 58
Deduplicated: 14
DB savedThis allows monitoring the system without server access.
Connecting to AI Automated Posting
The keyword collection process works independently but serves as input for the AI posting system:
Keyword collection
↓
Data storage in database
↓
AI generates content
↓
Image generation
↓
Automatic publishing
This creates a seamless workflow.
Practical Considerations for Real-World Use
When collecting news and keywords, consider:
Excluding outdated news
Avoiding promotional keywords
Removing duplicate articles
Skipping articles with identical titles
Eliminating meaningless words
Preventing excessive requests
These steps ensure a stable, reliable automation system.
Automation Lies in Connection
Many assume automation means just AI writing posts. In reality, it requires automating the entire process:
Find quality keywords,
Clean them,
Store them,
And deliver them to AI.
This project is a critical first step in building a complete system.
Conclusion
In this lesson, we built an automated system that collects and organizes news-based keywords for database storage. A stable AI posting system requires consistent keyword supply.
This project establishes the foundation for preparing data for AI use. In the next session, we'll implement Project 3: AI Image Generation and Automated Posting System, where we'll generate representative images for posts and automatically publish them to a live blog.
FAQ
Why is keyword selection important for AI blogging?
The right keywords determine search traffic and relevance; poor choices lead to low engagement.
How does the system avoid duplicate posts?
It removes recently used keywords (e.g., within 30 days) and deduplicates entries before storage.
Korean Source
This article is an English localization of a Korean-language report from BlogScore. Original: [Node.JS 자동화] 13강. 실전 프로젝트② 뉴스·키워드 자동 수집 시스템 만들기