Beam Us Up Crawler Updated v1.2.1 – Big Feature Update

  • Facebook
  • Twitter
  • Reddit

Download WindowsMac or Linux

Remember you need Java installed for Mac (how to run on mac guide) & Linux. As well as to uninstall any previous version.

Need Support?

Reply to a comment with your real email address in this post and I’ll get back to you.

Whats in the update?

So we’ve been busy working on updates to Beam Us Up crawler the updates are many many many!

Crawls are automatically saved

Crawling is saved periodically during a crawl.

If you crawl too many URLs and it freezes then and you restart the program it’ll load up from where you left off and you can complete it or delete it.

Right Click Copy

Lots of options to copy the tables.

Improvements in error filters

Color coded filters and cleaning up of the buttons

In and Outside of Sitemaps

See if urls you are seeing are inside or outside of sitemaps.

Easier to see in links and out links from a page

URL Search

Dark Mode

Bug Fixes and Minor Improvements

A lot of bug fixes and minor improvements!

Add comment

20 comments

  1. Ralf
    03 . Apr . 2024

    Hi William,
    thx a lot for the update! 🙂
    Will test it out in the next days.
    All the best!
    Ralf

    • An
      09 . Apr . 2024

      Have you tested it yet?

  2. Mario
    06 . Apr . 2024

    I tried to open the app in Mac with no success. I have the latest Java installed, but there is an error that doesn’t allow the app to open.

    • G
      15 . Apr . 2024

      Hi Mario,

      What error?

      Thanks.

    • Thomas
      17 . Apr . 2024

      Hey, I have same problem. Does anyone know how to fix it?

      • G
        22 . Apr . 2024

        The error I am guessing is that it does not open. This is because of Mac’s overly aggressive “security”/wanting developers to pay them money. We have applied to Apple and we are currently trying to sign the DMG file to allow it to open easily. However it is a bit of a pain especially when we ourselves are not Mac developers.

  3. Annie
    09 . Apr . 2024

    Thank you for the update!

  4. Adrian
    10 . Apr . 2024

    Hello,

    This is a nice upgrade!

    I’m trying to crawl a big web (in development), so I’m getting two errors.
    The first one:
    Exception in thread “AWT-EventQueue-0” java.lang.ArrayIndexOutOfBoundsException: Index 800 out of bounds for length 800
    at java.desktop/javax.swing.DefaultRowSorter.setModelToViewFromViewToModel(DefaultRowSorter.java:745)
    …………..

    The second one:
    ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console…
    Finished (killed)

    Regards

    • G
      15 . Apr . 2024

      Hi Adrian,

      Can you tell me what the operating system and website you are using is?

      (If you reply with the website here don’t worry I’ll remove it from your comments before publishing so it’ll remain private ;))

      Thanks.

      • Adrian
        15 . Apr . 2024

        Hello, I’m using it in Debian 12 and the website is in local with docker.
        The error comes because I got 144000 links and had not enough memory (16GB)… jeje So I think it was an error related no free memory (could not load it)

        It would be good to have a check options about the “extra” filters (because as I notice, are all data, not related). So, if I don’t want to check about the “URLs Not in Sitemap” in this round, will be faster, because there is no need to check it.

        Also, are you saving where am I comming from an URL? (as before, with a check option to do it, would be a good improvement). Sometimes is good to know which links points to “website.com/this-path”.

        Thanks for your effort!

        • G
          22 . Apr . 2024

          ” are you saving where am I comming from an URL? ” Yes but the crawl needs to finish or you need to stop it and then it will calculate this.

          114k urls is quite a lot. I doubt there are a lot of unique types of pages so I recommend you stop the crawl before it stops.

  5. Hielke Brandsma
    10 . Apr . 2024

    Hi Gui / William!

    I really appreciate all your time and effort building this!
    But .. I’m running into a small problem with retrieving content from our webshop The problem seems to have something to do with DNS. We use LightspeedHQ as webshop platform we are pointing the domainname to their servers, so from that point on, I’m unable to diagnose further. When I enter “our url” as URL, I only receive a HTTP 301 (Moved Permanently) message. I also found a CNAME in our DNS records: “7221.shops.oururl”, but this will be re-routed. I hope you have some answers for me, because I’m very curious 🙂

    Thanks in advance!
    Hielke

    • G
      15 . Apr . 2024

      Thanks for letting me know I get the same issue will let you know if and when it is fixed 🙂

    • G
      22 . Apr . 2024

      Fixed! Please uninstall and download again (assuming you are using windows if not then it isn’t fixed ;))

  6. Eugene
    22 . Apr . 2024

    Hello, tnx for app

    Some new futures:
    1) In Configuration check box – Do not crawl external domains
    2) And also will be good to have info about missing Alt and Title tags for images

    • G
      23 . Apr . 2024

      Hi Eugene,

      1. This already exists
      2. Good idea

      Thanks.

  7. Daniel
    30 . May . 2024

    Hello 🙂 Thanks for the very useful BeamUsUp crawlerl, I recommended it to some colleagues who are more at the beginners stage and get overwhelmed with all the options of SF. This was simpler and the colour coding helped. Just wondering, is there a way your tool can crawl JavaScript sites? As in when you switch Rendering to JS in the other tool, or is this not possible? Thanks again.

    • G
      28 . Jun . 2024

      Hi Daniel, thanks I appreciate it. Unfortunately adding that now is too complicated. Thanks

  8. Ado Cehic
    05 . Jul . 2024

    Hey! I’m running into an issue with the latest version on MacOS. I’ve got an Intel Mac running 13.6.7, and the Java (Running the latest available version 8_411) error I get is the following:

    Exception in thread “main” java.lang.UnsupportedClassVersionError: com/beamusup/webcrawler/WebCrawler has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:473)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:359)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
    at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:635)

    • G
      13 . Jul . 2024

      Thanks for letting us know we’re looking into getting a fully signed mac DMG file although apple make it extremely difficult especially for us non mac developers 🙁

Related

Beam Us Up Crawler Updated 2024 v1.1.1
5 months ago

After many attempts of finding someone to help me updated the crawler, I have finally ...

Web Based Crawler
2 years ago

Update: Unforuntately due to personal reasons the developer I was working on this with (who ...

Back in active development
4 years ago

Hello, So after around 5 years out of active development, I have decided to restart ...

Mac/Linux Version Support Discontinued
7 years ago

So currently we do not really support any of the software as it is largely ...