Beam Us Up Crawler Updated v1.3.0 – New features & Fixes

3 months ago 4 min read19 comments

Download Windows, Mac or Linux

Remember you need Java installed for Mac & Linux. As well as to uninstall any previous version.

Table of Contents

Installing on Mac Instructions

I unfortunately don’t have a Mac to test the Mac version however a user on Reddit has suggested to do the following: Open Terminal (edit locations as required): brew install openjdk sudo ln -sfn /opt/homebrew/opt/openjdk/libexec/openjdk.jdk /Library/Java/JavaVirtualMachines/openjdk.jdk java –add-exports java.desktop/com.apple.eawt=ALL-UNNAMED -jar /private/tmp/buu.jar

Need Support?

Reply to a comment with your real email address and real URL in this post and I’ll get back to you.

Whats in the update?

New Feature – Site Structure

You can now see the structure of the site your crawl all the folders etc.

New Feature – Issues

Now issues that are affecting the crawl like in this case cloudflare blocking access to the sitemap, appear int he new issues tab ont he right.

New Feature – Misc

Probably not issues just random things that could be interesting such as mailto links and file downloads.

General Improvements

Stop redirect loops, after a few loops the crawler will stop attempting to follow it.
Lots of bug fixes
Minor UX improvments

If you find bugs or have suggestions let me know in the comments!

Known Bug

If you have robots.txt parsing enabled in config and the website doesn’t have sitemaps then it will stop the crawl. This will be fixed in the next release (coming soon) which will also have more new features 🙂

Add comment

Cancel Reply

19 comments

Bryan Jarvis
08 . Jan . 2025

Sorry to report that the Windows version link goes to a 404 page.
- G
  08 . Jan . 2025
  
  Fixed 🙂
Mercenario
08 . Jan . 2025

Hi, and thanks for the update.

Is it possible to have the app describe the issue? See this screenshot on the sidebar of the app: https://i.imgur.com/Z5tIstU.png

I’m not sure what it means with that link.
- G
  09 . Jan . 2025
  
  Hi Mercenario,
  
  In that case it isn’t an issue it is just the “Misc” tab which just in that case puts files and stuff it can’t crawl but thinks are perhaps interesting for you to note. But it will improve and explain things better in the next update.
  
  Thanks.
Saiful
09 . Jan . 2025

hi want to ask, is the mac version for intel or arm? cause im using the mac m2 version
- G
  09 . Jan . 2025
  
  Hi Saiful,
  
  It is neither it is using Java so should work on anything. I however myself don’t have a Mac to test it. And you have to do it with the instructions above.
  
  Thanks
Allen
10 . Jan . 2025

Thank you for the update. Keep up the good work!
Jan
13 . Jan . 2025

Hey,

I run the software, tried it both on windows and mac – didn’t work for some reason – managed to install it on both machines ( both physical separate ones ), i press start, nothing happens – tried it with links that have a few things to check just to see whether i was doing it wrong, looks like something is broken and I can’t find what…

Thanks
- G
  13 . Jan . 2025
  
  Hey Jan,
  
  Unfortunately works fine for me and most others. Can you please give us a URL to check? Can you try https://beamusup.com as a URL?
  
  Thanks.
- G
  16 . Jan . 2025
  
  Hey Jan,
  
  I’ve got another theory can in the config you turn off “parse robots.txt”?
  
  Thanks.
macdet
14 . Jan . 2025

I need this beast for my broken / probs. Django is great BUT 🙁
Luciano
14 . Jan . 2025

Sorry, I pasted my post in the wrong place/version.

Hello, trying to start BeamUsUp in Linux Lubuntu 24.04 LTS but I keep getting the same runtime error

Exception in thread “AWT-EventQueue-0” java.lang.NullPointerException: Cannot read the array length because “” is null

Program starts but crawling is inhibited. I tried with different versions of java but nothing changes.
What can I do?
Thanks in advance
Ciao Luciano
- G
  15 . Jan . 2025
  
  Hi Luciano,
  
  Works fine for me on regular Ubuntu, so I am not exactly sure what to do.
  
  Sorry.
- G
  20 . Jan . 2025
  
  Hey Luciano,
  
  I made a mistake you should use the JAR file not he DEB file. Ive updated the links now.
  
  Thanks.
Matthew Leavitt
14 . Jan . 2025

Hello. Trying to use the software and assume I may not be able to as I am getting a “done with robots.txt” – tried deleting it from my subdirectory, but don’t have access to the primary website. Am I out of lucky?
- G
  15 . Jan . 2025
  
  Hey Matthew,
  
  I think i know the problem. Goto the configuration part and make sure “parse robots.txt” is not selected. Then it should work fine.
  
  Thanks
Florian
26 . Jan . 2025

Hi, Good software.

On previous release i have send you (on comment) a bug when i add “:8000” for start URL, this url is not kept for all scanned url and port is removed for relative URL.

The bug is already here in new version.

Thanks
Brian Metis
02 . Feb . 2025

Export of CSV is blank.
Only outputs two columns:

A B

Both for XLS and CSV formats.

Exporting 266,391 Links (rows)
1,048,576 Rows is Excel Max Count.
5,327,820 records total (approximate based on headers / columns)

File size is 1kb for second attempt and 0kb for others.
- G
  03 . Feb . 2025
  
  Hi Brian,
  
  The CSV export is only for whatever is shown in the bottom half of the tool (individual URL results) so you have to select a URL for it to work, which is why it might be empty.
  
  If it is not working for you for the XLS I cannot replicate it however to be honest, I’ve never crawled such a large site can you send me the URL?
  
  Thanks.