How to extract javascript dictionaries scrapy

Why would you want to do that?

Well, if you are web scraping using Python, and Scrapy for instance, you may need to extract reviews, or comments that are loaded from JavaScript. This would mean you could not use your css or xpath selectors like you can with regular html.

Parse

Instead, in your browser, check if you may be able to parse the code, beginning with ctrl + f, and “json” and track down some JSON in the form of a python dictionary. You ‘just’ need to isolate it.

view-source to find occurrences of “JSON” in your page

The response is not nice, but you can gradually shrink it down, in Scrapy shell or python shell…

Figure 1 – The response

Split, strip, replace

From within Scrapy, or your own Python code you can split, strip, and replace, with the built-in python commands until you have just a dictionary that you can use with json.loads.

x = response.text.split('JSON.parse')[3].replace("\u0022","\"").replace("\u2019m","'").lstrip("(").split(" ")[0].strip().replace("\"","",1).replace("\");","")

Master replace, strip , and split and you won’t need regular expressions!

With the response.text now ready as a JSON friendly dictionary you can do this:

import json
q = json.loads(x)

comment = (q[‘doctor’][‘sample_rating_comment’])

comment.replace(“\u2019″,”‘”)
print(comment)

The key thing to remember to use when parsing the response text is to use the index, to pick out the section you want, and then make use of “\” backslash to escaped characters when you are working with quotes, and actual backslashes in the text you’re parsing.

Figure 2 – The parsed response

Conclusion

Rendering to HTML using Splash, or Selenium, or using regular expressions are not always essential. Hope this helps illustrate how you can extract values FROM a python dictionary FROM json FROM javascript !

You may see a mass of text on your screen to begin with, but persevere and you can arrive at the dictionary contained within…

Password Hashing with bcrypt (easiest explanation)

#node #javascript #react #angular #AI, #Analytics, #BigData, #CloudComputing, #DataScience, #DataScientist, #IoT, #Java, #JavaScript, #Linux, #MachineLearning, #Programming, #Python, #ReactJS, #RStats, #Serverless, #TensorFlow

We use bcrypt to hash our passwords. But how to use it? We generally do 2 basic things with bcrypt.

• hash a password (I mean, when signing up, we hash the password input and then save this hashed password instead of the plain password on our database)

• verify password (I mean, when logging in, compare the plain password input with the hashed password that we saved)

The SIMPLEST WAY TO USE BCRYPT

• Hash a password

//it creates the hashed password. Save this hashedPassword on your DB
const hashedPassword = bcrypt.hashSync(yourPasswordFromSignupForm, bcrypt.genSaltSync());

now save this hashedPassword on your Database.

• Verify Password

const doesPasswordMatch = bcrypt.compareSync(yourPasswordFromLoginForm, yourHashedPassword)

doesPasswordMatch is a bolean. If the passwords match, it'll be true, else false.

COMPLETE GUIDE FOR USING BCRYPT

First, type this on your terminal to install the bcryptjs package
npm install bcryptjs

Now we are ready to use it.

Step 0.

Create your user model. In this case we are going to keep it simple. our model will only have email and password fields.

Step 1 (USING BCRYPT TO SAVE HASHED PASSWORD ON DB FOR SIGN UP).

const router = require('express').Router();
const User = require('YOUR_USER_MODEL');


const bcrypt = require('bcryptjs')


router.post('/signup', async (req, res)=>{
  // these emailFromSignupForm and passwordFromSignupForm are coming from your frontend
  const { emailFromSignupForm, passwordFromSignupForm } = req.body;

 //creating a new user on our database
  const newUser = await User.create({
  email: emailFromSignupForm,
  hashedPassword: bcrypt.hashSync(passwordFromSignupForm, bcrypt.genSaltSync()),
});

//sending back the newUser to the frontEND
res.json(newUser);


})


module.exports = router;

This is a demo code of how to use bcrypt to hash the password and save the hashed password.

Step 2 (USING BCRYPT TO COMPARE PASSWORDS FOR LOG IN).

const router = require('express').Router();
const User = require('YOUR_USER_MODEL');


const bcrypt = require('bcryptjs')


router.post('/login', async (req, res)=>{
  // these emailFromLoginForm and passwordFromLoginForm are coming from your frontend
  const { emailFromLoginpForm, passwordFromLoginForm } = req.body;

  //find a user from the database with your emailFromLoginForm
 const existingUser = await User.findOne({ email: emailFromLoginForm });

//if no user found
if(!existingUser) return res.json({ msg: `No account with this email found` })

//if the user is found, I mean if the user is on our database, compare the passwordFromLoginForm with the hashedPassword on our database to see if the passwords match (bcrypt will do this for us)
const doesPasswordMatch = bcrypt.compareSync(passwordFromLoginForm, existingUser.hashedPassword); //it wii give you a boolean, so the value of doesPasswordMatch will be a boolean

//if the passwords do not match
if(!doesPasswordMatch) return res.json({ msg: `Passwords did not match` });

//if the passwords match, send back the existingUser to the frontEND
res.json(existingUser);
}


})


module.exports = router;

This is a demo code of how to use bcrypt to compare and verify passwordFromYourLoginForm with the hashedPassword saved on your Database.

This is ONLY a demo of how to use bcrypt. Hope it helps.

If you have any Questions or If you are stuck

Feel free to reach out to me. 

I'd LOVE to be your friend, feel FREE to reach out to me!!

If this blog was helpful to you,

PLEASE give a LIKE and share,

it'd mean a lot to me. Thanks

Reset radio selection from click [ #javascript, #html]

Your attempts to reset the form to its initial state are actually working but the problem is that, because you are submitting the form every time you click on an input, the page and, therefore, the form are being reloaded and the last clicked input now has an initial state of checked. So, what you need to do is to loop through all the checked inputs and "uncheck" them. Here's a trimed down example of one way of doing so:

var inputs=document.querySelectorAll("input[type=radio]:checked"),
    x=inputs.length;
document.querySelector("button[type=reset]").addEventListener("click",function(event){
    while(x--)inputs[x].checked=0;
    event.preventDefault();
},0);

<form>
    <input id="option1a" name="option1" type="radio" value="a"><label for="option1a">Option 1A</label>
    <input checked id="option1b" name="option1" type="radio" value="b"><label for="option1b">Option 1B</label>
    <input id="option1c" name="option1" type="radio" value="c"><label for="option1c">Option 1C</label>
    <hr>
    <input id="option2a" name="option2" type="radio" value="a"><label for="option2a">Option 2A</label>
    <input checked id="option2b" name="option2" type="radio" value="b"><label for="option2b">Option 2B</label>
    <input id="option2c" name="option2" type="radio" value="c"><label for="option2c">Option 2C</label>
    <hr>
    <button type="reset">Clear</button>
</form>