Get content material from nearly any website to your WordPress weblog, mechanically!
Docs
| Demo
| Website
| Join our Discord server! (New)
FOR WHAT IT CAN BE USED
- Create a private website which collects information, posts, and many others. out of your favourite websites to see them in a single place
- Use it with WooCommerce to gather merchandise from purchasing websites
- Acquire merchandise from affiliate applications to earn money
- Acquire posts to create a check setting on your plugin/theme
- Acquire plugins, themes, apps, pictures from different websites to create a set of them
- Maintain observe of rivals
- You may think about something. The web is stuffed with contents
Before you purchase, be sure you do the next:
-
Watch the quick start video and use the plugin within the
demo. You may also watch the opposite
video tutorials to learn to use
the plugin. There are additionally many guides
explaining the way to do sure issues with the plugin. -
Be sure the plugin can retrieve the info from the positioning you need to crawl by following the directions in
Can I get content from X site? FAQ. -
In case you are nonetheless undecided if the plugin can retrieve content material from a particular website, ask us within the feedback
part. -
You may examine the FAQs
if in case you have any questions. If the reply to your query isn’t there, you’ll be able to at all times ask us within the feedback
part.
30-SECOND* SITE SETUP WITH CONFIG HELPER
*Config Helper is displayed when creating a brand new website. Its objective is to hurry up the preliminary setup for a website.
Though Config Helper works for a lot of websites, not each website might be configured this simply, which means that sure
websites might require a handbook and extra elaborate setup. Even for these websites, you’ll be able to attempt to create the essential
configuration with Config Helper after which modify the positioning settings later. Moreover, it’s doable to exit
Config Helper any time or disable Config Helper fully.
QUICK START
HOW IT WORKS
It’s all about CSS selectors and you may learn to use them in minutes by watching the introduction tutorial. The plugin’s Visible Inspector instrument additionally helps you discover CSS selectors simply by clicking onto the weather within the goal websites. Right here is the gist of it:
WHAT WP CONTENT CRAWLER CAN DO
Right here is the listing of some options of WP Content material Crawler. To find out about the entire options, please see the options desk under.
HTML code of social media posts present within the goal publish web page is mechanically transformed to brief codes. By this
approach, they’re displayed on the entrance finish of your web site accurately. Moreover, all of the iframe components are
transformed to a brief code. An iframe brief code is displayed provided that its supply is trusted. If the supply of an
iframe isn’t trusted by default, you’ll be able to manually add a trusted area in order that the iframe is displayed. With this
technique, you’ll be able to show media from third-party websites securely. Web sites whose media are embedded mechanically
embody Instagram, Imgur, YouTube, Vimeo, TikTok, Kickstarter, Twitter, Pinterest, and so forth.
SEE IT IN ACTION, LEARN IN MINUTES
 |
WP Content material Crawler introduction video (English) |
 |
WP Content material Crawler introduction video (Turkish) |
VIDEO TUTORIALS
 |
Fast Begin Information |
 |
Utilizing CSS Selectors in WP Content material Crawler |
 |
HTML & CSS Selectors |
 |
Utilizing brief codes to put any information anyplace within the publish |
 |
Save pictures as WooCommerce product gallery |
 |
Save arcade video games |
MAIN FEATURES
| Save each publish element Title, excerpt, content material, tags, classes, slug, date, customized meta, taxonomies, meta key phrases, meta description, featured picture, publish pictures, standing… Simply every part. |
Visible Inspector Simply click on to a component to search out its CSS selector. You may also get different CSS selectors that you just may be curious about. There isn’t any want to go away your admin panel anymore. |
|
| Crawl (scrape, seize, save) posts After the settings are configured, the plugin finds URLs of the posts and crawls them mechanically within the background. |
Recrawl (replace) posts |
|
| Delete posts You need to delete outdated crawled posts? The plugin can delete them mechanically. |
Management scheduling |
|
| Save classes The goal class doesn’t exist in your website? No drawback. The plugin can create the goal classes for you. Simply outline the CSS selectors that discover class names. They will even be created as subcategories. |
Save slugs (permalink) You may outline the permalink of the posts. You will get the permalink from the goal website, enter customized textual content, and even create templates for the slugs through the use of brief codes. |
|
| Save taxonomies Save taxonomy values by retrieving them from the goal website or coming into manually. Saving particulars of customized publish sorts is less complicated than ever. |
Save posts into customized classes |
|
| Customized publish meta Save something as customized publish meta. You should utilize a CSS selector or simply sort the worth. |
Content material templates |
|
| Various selectors You may write different selectors to get the info even when the goal website has publish pages designed in a different way from one another. |
Discover and substitute something |
|
| Paginated posts Goal publish has multiple web page? No worries. It can save you paginated posts as nicely. |
Record sort posts Some websites create posts with a listing inside. You may extract the listing from the publish, create a template that must be utilized to every listing merchandise and even reverse the listing. |
|
| Take away pointless components Generally it is advisable to do away with some components, similar to ads, feedback, you identify it. Simply write its CSS selector and it’s eliminated. |
Robotically insert class URLs Goal website has lots of of classes? Piece of cake. Simply write the CSS selector and the plugin will insert them for you. |
|
| Publish sorts Set publish sort. It may be a publish, a web page, a product, or some other publish sort accessible in your WordPress set up. |
Take away hyperlinks You may take away hyperlinks from the publish. Simply examine the checkbox and the hyperlinks are gone. That simple. |
|
| Password safety You may set a password for the posts to point out them solely to the customers who’ve the password. |
Notes You may add notes for your self to remind you issues in regards to the website. CSS selectors, TODO listing, something. |
|
|
Take a look at every part on the fly |
Take a look at all of the settings of a website directly Utilizing the tester, you’ll be able to check all choices you configured within the website settings to verify every part works as you need earlier than enabling automated crawling. |
|
| Instruments Utilizing the instruments, it can save you posts manually with their URL, recrawl posts with their ID or delete already-saved URLs. |
Customized basic settings for every website You may present customized basic settings for every publish to override them and make them appropriate for a website. |
|
| Publish standing You may instantly publish the saved posts or hold them as draft to examine them earlier than publishing. |
Save all pictures in publish content material Saving all pictures within the content material of the publish is as simple as checking a single checkbox. |
|
|
Save pictures as gallery |
Any information as brief code Get something from goal web page as a brief code and use the brief codes within the plugin’s templates to put any information anyplace you need. |
|
| Proxy Use a proxy or proxies to get content material from the websites to which your IP doesn’t have entry. |
Cookies Connect cookies, similar to session cookies, to every request. By this manner, for instance, you’ll be able to crawl the goal website as if you’re logged in. |
|
| Crawl as many posts as you need You may set what number of occasions publish crawling or URL assortment CRON occasions ought to run. By this manner, you’ll be able to, e.g., save 100 posts each minute. Simply watch out and contemplate your server’s capability. |
Electronic mail notifications Set CSS selectors whose values shouldn’t be empty for class and publish pages. When an empty worth is discovered utilizing these selectors, you may get an e-mail notification. |
|
| Get information from JSON While you allow JSON parsing for a CSS selector, you may get the values from the JSON simply. |
Superior HTML manipulations |
|
|
Automated translation |
Automated spinning Use spinning to mechanically rewrite crawled posts’ contents to enhance SEO. The plugin presently implements Spin Rewriter API and Turkce Spin API, that are paid providers. You may go to their web site to study the pricing particulars. |
|
|
Duplicate publish examine |
Scheduled posts You may add/take away minutes to/from the publish date. By this manner, you’ll be able to schedule publish publishing. |
|
| Save WooCommerce merchandise Save worth, stock, delivery, attributes, and superior choices. It can save you the product as a easy or an exterior product. You may also set downloadable file choices and outline the product as digital. The choices can be found for WooCommerce variations higher than or equal to three.3. |
Choices field You may have the management! Outline many choices for the values discovered by a CSS selector. The choices embody find-replace, calculation, template, and JSON parsing settings. You may simply import/export the choices outlined within the choices packing containers as nicely. |
|
| Deal with information like a professional Rename, copy, and transfer saved information simply. You may also outline title, description, caption, and alt texts for the saved media information utilizing templates through which you should utilize any brief code. It’s also doable to present random names to the saved information. |
Deal with iframes and scripts like a professional WordPress doesn’t enable exhibiting iframes and scripts since they pose a safety danger. You may flip iframe and script HTML components into brief codes by simply checking a checkbox. The brief code will present iframes and scripts from the allowed supply domains outlined by you. |
|
| Fast save With fast save button, it can save you the settings far more rapidly. No want to attend for web page to reload. |
Common expressions Outline common expressions in find-replace choices to find-replace something. You may also use delimiters and modifiers to match extra exactly. |
|
| Save “srcset” attributes When different sizes of the saved pictures can be found, the plugin assigns them into srcset attribute of img components in order that your pages will load quicker in numerous display sizes. |
Save “alt” and “title” attributes |
|
| Warnings Be taught when there’s a drawback. The plugin will present you the small print of the error so that you could repair it immediately. |
Deal with character encoding issues |
|
| Navigate between settings simply Repair navigation to the highest! The plugin shops the place you had been earlier than switching to a brand new tab and restores your earlier location while you activate that tab once more. No extra getting misplaced among the many settings. |
Guide crawling instrument With handbook crawling instrument, save a number of posts by coming into their URLs. You may also enter class URLs in order that the instrument can get publish URLs from there. Furthermore, you’ll be able to set it to crawl a number of posts on the identical time. |
|
|
Add URLs to the database |
Allow/disable automated crawling for a particular website You may allow or disable automated crawling for every website individually. |
|
| Import/export You may import and export website settings simply. Simply copy and paste the code created by the plugin. |
Limitless Add limitless websites to the plugin and activate what number of of them you need. |
|
|
Detailed dashboard |
Get updates out of your admin panel You may replace the plugin with only one click on each time an replace is prepared. Simply go to your updates web page in your admin panel. |
|
| Use probably the most safe PHP The plugin helps the most recent variations of PHP. |
Use probably the most fashionable browsers The plugin helps Chrome, Firefox, Safari, Opera, and Edge. |
|
| Interactive guides Interactive guides present you the way to configure settings to realize sure issues, step-by-step, like a residing documentation. You can begin these guides everytime you need. You may even begin them from a particular step. |
On-line documentation You may examine the net documentation everytime you really feel a necessity. |
|
| Fast guides proper subsequent to the settings Every setting within the plugin has a fast information that may assist you perceive what every setting does. |
Video tutorials Watch video tutorials to simply learn to use the plugin. |
|
| Able to translate You may translate the plugin into your individual language utilizing Poedit. |
Filters |
|
| Use OpenAI GPT (ChatGPT) You should utilize OpenAI GPT fashions to alter the title, content material, tags, file names, and extra. You should utilize GPT-3.5 and GPT-4. With the superior brief code builder, you should utilize the chat, full, edit, and insert modes. To study extra, watch this video! |
Convert JSON to HTML |
|
| Embed social media posts mechanically Posts from 70+ web sites together with Instagram, Fb, Amazon, YouTube, Twitter, Scribd, Vimeo, Pinterest, Spotify, Meetup, and plenty of others are transformed to embed brief codes mechanically. By this manner, they’re displayed securely on the entrance finish of your web site! |
Make customized requests You may make customized requests to different APIs or web sites and embody the response into the present web page. Watch this video to see how highly effective this function is! This additionally makes it doable to retrieve content material added to the web page by way of AJAX. |
| Necessities | PHP >= 8.1, json, mbstring, curl, dom, fileinfo, WP-Cron. These are already accessible in most hosts. Even when the extensions will not be already lively, most internet hosting websites allow you to allow these from their management panel. See the documentation for extra info. |
| Examined with WP variations | 6.6, 6.5, 6.4, 6.3, 6.2, 6.1, 6.0, 5.9 |
| Examined with WooCommerce variations | 9.2, 9.1, 9.0, 8.7, 8.2, 7.9, 7.7, 7.5, 7.3, 6.9 |
| Languages | English, Türkçe |
| Shortcomings | The plugin can’t retrieve content material that’s created through the use of JavaScript. For extra info, please see Can I get content from X site?. |
HAPPY CUSTOMERS 
WHY WP CONTENT CRAWLER
Issues with crawling an internet site
- Not a simple process, requires superior programming expertise
- Each web site is completely different and wishes tailor-made crawling implementation
- Not simply each web site is completely different, but additionally pages of a single web site can differ
- Pages and their supply codes must be investigated intensively to give you a crawling plan
- Figuring out the way to save sure info in a particular place in WordPress requires information in regards to the inside construction of WordPress and the way WordPress works
- If sure info must be saved into a particular subject outlined by a third-party plugin, one ought to modify the crawling implementation after researching for hours about the way to save that info
- One ought to learn about how HTML works and the way to extract sure elements from HTML code
- One ought to deal with all doable inconsistencies that may be within the supply codes of internet sites to offer a sturdy resolution that may hold working
- What if the posts must be shared in common time intervals?
- What if you wish to crawl new posts added to an internet site after a while?
- What about translating the posts from one language to a different?
- What if the posts must be paraphrased to offer a greater SEO for the web site?
- What if some info shouldn’t be retrieved?
- What if sure info must be modified to make it appropriate on your website?
- What if one other website must be crawled, not only one?
- What if that different website wants a unique crawling plan?
- What if it is advisable to login to the web site to crawl it?
- What if the web site modifications its supply code?
- What if you wish to replace the crawled posts by recrawling them from the unique web site?
- What if you wish to be certain that if the knowledge is retrieved precisely as you need it earlier than mechanically posting the posts to your web site?
- What if you wish to guarantee your website’s safety by ensuring no malicious-code-executing code results in your website?
- And lots of extra what-ifs that you just may not even think about except you come throughout them
Our imaginative and prescient and mission
We consider that sturdy, dependable, and automated crawling capabilities must be accessible for anybody. We need to democratize this subject by letting anybody have these capabilities, not simply builders. With this objective, we intention to offer a plugin that you’ll fall in love with and really feel at residence when utilizing it. To let it accessible by anybody, we make the plugin low-cost and easy-to-use. We don’t implement the options simply to make gross sales. We plan and execute for the long run. We at all times hearken to your suggestions and make required modifications accordingly. We expect that WordPress plugins must be developed with enterprise-level care. So, we intensively check the plugin earlier than every launch with automated end-to-end UI assessments, presently over 1700 assessments, that run in many alternative environments within the cloud for a complete of over 40 hours to make sure the plugin is appropriate along with your server and WordPress environments and also you, our beneficial clients, get the standard and reliability you deserve.
How we resolve these issues
Now we have been creating WP Content material Crawler for nearly 4 years such that we now have come throughout nearly all of the what-ifs. Working with our clients and listening to their wants, we offer sturdy and dependable options to those issues. We consider that one ought to simply present from which website the knowledge must be retrieved and what info must be retrieved from that web site after which begin crawling that website, with out worrying in regards to the advanced behind-the-scenes operations.
To make it accessible to anybody, we offer an in depth on-line documentation that incorporates not simply the outline of the settings however the way to use the settings to realize your targets. Generally you may not really feel like studying the documentation. We additionally present interactive step-by-step guides which might be accessible within the plugin, only one click on away. You can begin the interactive guides exhibiting you step-by-step how you are able to do sure issues any time and from any step you need.
One of the distinctive options of WP Content material Crawler is the power to check nearly any configuration. By this manner, you’ll not come throughout any surprises after you allow automated crawling. When testing, the errors associated to your settings are proven so that you could repair them earlier than they trigger any issues.
WP Content material Crawler has so many options that even we have no idea what number of of them are there. You may mechanically crawl, replace, and delete the posts, you’ll be able to translate posts, spin posts, you’ll be able to even outline what fields must be translated or spun if you don’t want all of them modified. You’ll find-replace nearly something. You may assign some info from the goal publish to a brief code and place that info anyplace within the publish. It can save you WooCommerce merchandise. It can save you particulars for third-party plugins that we don’t even know they exist. The options of the plugin are designed such that you just really feel that you’re in management while you use them. We make them as versatile as doable to make them suit your wants. When designing new options, we at all times remember the fact that you may want a extra superior model of that function and we design the options accordingly. We make sure that the options and the complete code of the plugin are maintainable and extendable in order that we are able to at all times enhance the plugin.
CHANGELOG
Changelog is kept in the documentation site. Click here to see the changelog.
