Welcome to our Coding with python Page!!! hier you find various code with PHP, Python, AI, Cyber, etc ... Electricity, Energy, Nuclear Power

Showing posts with label Tutorial. Show all posts
Showing posts with label Tutorial. Show all posts

Monday, 1 May 2023

Watershed delimitation/delineation based on a DEM using ArcGIS Desktop

Here is a step-by-step tutorial on how to create a watershed delimitation based on a Digital Elevation Model (DEM) using ArcGIS Desktop:


Step 1: Obtain DEM data

1. Download DEM data for your area of interest from a reliable source such as the USGS Earth Explorer or the National Elevation Dataset.

2. Import the DEM data into ArcGIS by opening the ArcMap application and selecting File > Add Data > Add Raster.

Step 2: Preprocessing the DEM data

1. Ensure that the DEM data is projected in a coordinate system suitable for the area of interest by checking the data properties under the Layer Properties > Source tab.

2. Fill any sinkholes in the DEM data using the Fill tool. This ensures that all water flows are accounted for in the analysis. To do this, go to Spatial Analyst Tools > Hydrology > Fill and set the input raster to your DEM data.

3. Generate a flow direction grid using the Flow Direction tool. This tool assigns a direction of steepest descent for each cell in the DEM. To do this, go to Spatial Analyst Tools > Hydrology > Flow Direction and set the input raster to the filled DEM data.

4. Generate a flow accumulation grid using the Flow Accumulation tool. This tool counts the number of cells that contribute to each cell in the DEM. To do this, go to Spatial Analyst Tools > Hydrology > Flow Accumulation and set the input raster to the flow direction grid.

Step 3: Create the watershed delimitation

1. Determine the pour point location for the desired watershed. This is the location where water flows out of the watershed. You can select this point manually or use another method to derive it from other data layers.

2. Create a raster of the pour point location by converting it to a raster using the Feature to Raster tool. This converts the point location to a cell with a value of 1 in the raster.

3. Generate a watershed grid using the Watershed tool. This tool creates a raster grid where each cell is assigned to the watershed that drains into the pour point location. To do this, go to Spatial Analyst Tools > Hydrology > Watershed and set the input flow direction raster to the flow direction grid, the input pour point raster to the pour point location raster, and the input flow accumulation raster to the flow accumulation grid.

Step 4: Refine the watershed boundary

1. Convert the watershed raster to a vector using the Raster to Polygon tool. This tool converts the raster cells to polygons that define the watershed boundary.

2. Clip the vector polygon to the desired boundary using the Clip tool. This tool removes any portions of the watershed that extend beyond the desired boundary.

Step 5: Display and analyze the results

1. Add the clipped watershed boundary to your ArcMap project and visualize it using the symbology and labeling tools.

2. Perform any further analysis on the watershed boundary as needed, such as calculating the area, length, and slope of the watershed, or overlaying other data layers such as land use or soil data.

That's it! By following these steps, you should be able to create a watershed delimitation based on a DEM using ArcGIS Desktop.

These steps could be also introduced in a ModelBuilder



Saturday, 29 April 2023

A high-level tutorial on how to implement face detection and recognition using Node.js.

A high-level tutorial on how to implement face detection and recognition using Node.js.




  1. Install dependencies:

First, you'll need to install a few dependencies, including OpenCV, Node.js, and the OpenCV bindings for Node.js. You can find installation instructions for these dependencies online.

  1. Set up face detection:

To detect faces in an image, you can use the Haar Cascade Classifier algorithm provided by OpenCV. This algorithm works by analyzing the image at different scales and searching for patterns that resemble faces.

Here's an example code for face detection in Node.js using OpenCV:

javascript
const cv = require('opencv4nodejs'); const img = cv.imread('path/to/image.jpg'); const classifier = new cv.CascadeClassifier(cv.HAAR_FRONTALFACE_ALT2); const faces = classifier.detectMultiScale(img).objects;

In this example, we load an image using the imread() function from OpenCV. Then, we create a CascadeClassifier object using the HAAR_FRONTALFACE_ALT2 classifier, which is optimized for face detection. Finally, we use the detectMultiScale() function to search for faces in the image and return their coordinates as an array of rectangles.

  1. Set up face recognition:

To recognize faces in an image, you can use a machine learning algorithm that has been trained on a set of known faces. One popular algorithm for face recognition is called the FaceNet model, which uses a neural network to generate a high-dimensional vector representation of each face, called an embedding. These embeddings can be compared to determine if two faces belong to the same person.

Here's an example code for face recognition in Node.js using the FaceNet model:

javascript
const faceapi = require('face-api.js'); const canvas = require('canvas'); const fs = require('fs'); const img = await canvas.loadImage('path/to/image.jpg'); const modelPath = 'path/to/facenet-model.json'; await faceapi.nets.faceRecognitionNet.loadFromDisk(modelPath); await faceapi.nets.faceLandmark68Net.loadFromDisk(modelPath); const detections = await faceapi.detectAllFaces(img).withFaceLandmarks().withFaceDescriptors(); const descriptors = detections.map(d => d.descriptor); const labeledDescriptors = [ new faceapi.LabeledFaceDescriptors('Person 1', descriptors[0]), new faceapi.LabeledFaceDescriptors('Person 2', descriptors[1]) ]; const recognizer = new faceapi.FaceRecognizer(); recognizer.train(labeledDescriptors); const result = recognizer.recognizeSingleDescriptor(descriptors[2]); console.log(result.label);

In this example, we use the face-api.js library, which provides an easy-to-use interface for the FaceNet model. First, we load an image using the loadImage() function from the canvas library. Then, we load the FaceNet model from disk using the loadFromDisk() function from face-api.js. We use the detectAllFaces() function to detect all faces in the image and generate their embeddings using the withFaceDescriptors() function. We then create an array of labeled embeddings using the LabeledFaceDescriptors class. Finally, we train a FaceRecognizer object on the labeled embeddings using the train() function, and use the recognizeSingleDescriptor() function to recognize a single face in the image and return its label.

Note that this is a simplified example, and in practice, face recognition can be more complex and may require more advanced techniques such as face alignment and normalization.

In summary, face detection and recognition can be implemented using Node.js and various libraries such as OpenCV and face-api.js. Face detection involves searching for patterns that resemble faces

Monday, 15 July 2019

Doxygen Tutorial

DOXYGEN
Tutorial de documentação de códigos usando a ferramenta Doxygen no Windows
O QUE É:
O Doxygen é um programa que gera a documentação de um programa a partir da análise de um código
escrito em C (ou linguagens derivadas como C++, Java ou C#). Nesses códigos são reconhecidas
estruturas de dados, funções e comentários feitos com uma sintaxe especial.
COMO FUNCIONA:
O Doxygen analisa o código e gera um arquivo no formato CHM (Compiled HTML Help), mas para isso
você deve instalar o Microsoft HTML Help que pode ser baixado aqui ou aqui
.
COMENTANDO UM CÓDIGO:
Para usar o Doxygen, você deve comentar seu código de maneira diferente da usual (// ou /**/).
Os exemplos mais comuns de comentários reconhecidos pelo doxygen são:
/// COMENTÁRIO
/**
* COMENTÁRIO
*/
//! COMENTÁRIO
int variável; /*!< COMENTARIO SOBRE A VARIAVEL */
Mais tipos (comentários de funções, por exemplo) podem ser facilmente encontrados na internet.
CONFIGURANDO O DOXYWIZARD:
Em Wizard:
Project:
•
Project name: nome do seu projeto.
•
Source Code Directory: diretório onde estão as fontes.
•
Destination Directory: diretório onde será colocada a documentação.
Mode:
•
Selecione ALL ENTITIES, INCLUDE CROSS-REFERENCED SOURCE e OPTIMIZE FOR C OUTPUT.
Output:
•
Em HTML, selecione PREPARE FOR COMPRESSED HTML (.CHM).
•
Desmarque os demais formatos.
Diagrams:
•
Selecione NO DIAGRAMS.
Em Expert:
Project:
•
Selecionar OUTPUT_LANGUAGE Brazilian.
•
Desmarcar FULL_PATH_NAMES.
Build:
•
marcar INTERNAL_DOCS.
Input:
•
INPUT_ENCODING: trocar UTF-8 por char.
HTML:
•
CHM_FILE: colocar o nome do arquivo CHM a gerar.
Salvar as configurações (Save) e clicar em “Run Doxygen” na aba “Run”.
CRIANDO O ARQUIVO .CHM
Se tudo der certo, será criada uma pasta “html” no seu diretório de saída e dentro dessa pasta está um
arquivo .hhp. Duplo clique para abrir o Help Workshop. Selecione “File Compile” ou clique no ícone de
“moedor de carne”. No diretório “html” será criado um arquivo .chm. Abra o arquivo para ver o
resultado final. Essa é a documentação do seu código.
FIGURA 1 - HTML Help Workshop
FIGURA 2 - RESULTADO FINAL

Monday, 6 August 2018

Tutorial HANDLING JAVASCRIPT IN SCRAPY WITH SPLASH

A common roadblock when developing spiders is dealing with sites that use a heavy amount of JavaScript. Many modern websites run entirely on JavaScript, and require scripts to be run in order for the page to render properly. In many cases, pages also present modals and other dialogues that need to be interacted with to show the full page. In this post we’re going to show you how you can use Splash to handle JavaScript in your Scrapy projects.

What is Splash?

Splash is our in-house solution for JavaScript rendering, implemented in Python using Twisted and QT. Splash is a lightweight web browser which is capable of processing multiple pages in parallel, executing custom JavaScript in the page context, and much more. Best of all, it’s open source!

Setting Up Splash

The easiest way to set up Splash is through Docker:
$ docker pull scrapinghub/splash
$ docker run -p 5023:5023 -p 8050:8050 -p 8051:8051 scrapinghub/splash
Splash will now be running on localhost:8050. If you’re using a Docker Machine on OS X or Windows, it will be running on the IP address of Docker’s virtual machine.
If you would like to install Splash without using Docker, please refer to the documentation.

Using Splash with Scrapy

Now that Splash is running, you can test it in your browser:
http://localhost:8050/
splash-amazon
On the right enter a URL (e.g. http://amazon.com) and click 'Render me!'. Splash will display a screenshot of the page as well as charts and a list of requests with their timings. At the bottom you should see a text box containing the rendered HTML.

Manually

You can use Request to send links to Splash:
req_url = "http://localhost:8050/render.json"
body = json.dumps({
    "url": url,
    "har": 1,
    "html": 0,
})
headers = Headers({'Content-Type': 'application/json'})
yield scrapy.Request(req_url, self.parse_link, method='POST',
                                 body=body, headers=headers)
If you’re using CrawlSpider, the easiest way is to override the process_links function in your spider to replace links with their Splash equivalents:
def process_links(self, links):
    for link in links:
        link.url = "http://localhost:8050/render.html?" + urlencode({ 'url' : link.url })
    return links

Scrapy-Splash (recommended)

The preferred way to integrate Splash with Scrapy is using scrapy-splash. See here for why it's recommended you use the middleware instead of using it manually. You can install scrapy-splash using pip:
pip install scrapy-splash
To use scrapy-splash in your project, you first need to enable the middleware:
DOWNLOADER_MIDDLEWARES = {
    'scrapy_splash.SplashCookiesMiddleware': 723,
    'scrapy_splash.SplashMiddleware': 725,
    'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}
The middleware needs to take precedence over HttpProxyMiddleware, which by default is at position 750, so we set the middleware positions to numbers below 750.
You then need to set the SPLASH_URL setting in your project's settings.py:
SPLASH_URL = 'http://localhost:8050/'
Don’t forget, if you’re using a Docker Machine on OS X or Windows, you will need to set this to the IP address of Docker’s virtual machine, e.g.:
SPLASH_URL = 'http://192.168.59.103:8050/'
Enable SplashDeduplicateArgsMiddleware to support cache_args feature: it allows to save disk space by not storing duplicate Splash arguments multiple times in a disk request queue. If Splash 2.1+ is used the middleware also allows to save network traffic by not sending these duplicate arguments to Splash server multiple times.
SPIDER_MIDDLEWARES = {
    'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
}
Scrapy currently doesn’t provide a way to override request fingerprints calculation globally, so you will also have to set a custom DUPEFILTER_CLASS and a custom cache storage backend:
DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'
HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage'
If you already use another cache storage backend, you will need to subclass it and replace all calls to scrapy.util.request.request_fingerprint with scrapy_splash.splash_request_fingerprint.
Now that the Splash middleware is enabled, you can use SplashRequest in place of scrapy.Request to render pages with Splash.
For example, if we wanted to retrieve the rendered HTML for a page, we could do something like this:
import scrapy
from scrapy_splash import SplashRequest

class MySpider(scrapy.Spider):
    start_urls = ["http://example.com", "http://example.com/foo"]

    def start_requests(self):
        for url in self.start_urls:
            yield SplashRequest(url, self.parse,
                endpoint='render.html',
                args={'wait': 0.5},
            )

    def parse(self, response):
        # response.body is a result of render.html call; it
        # contains HTML processed by a browser.
        # …
The ‘args’ dict contains arguments to send to Splash. You can find a full list of available arguments in the HTTP API documentation. By default the endpoint is set to ‘render.json’, but here we have overridden it and set it to ‘render.html’ to provide an HTML response.

Running Custom JavaScript

Sometimes you may need to press a button or close a modal to view the page properly. Splash lets you run your own JavaScript code within the context of the web page you’re requesting. There are several ways you can accomplish this:

Using the js_source Parameter

You can use the js_source parameter to send the JavaScript you want to execute. The JavaScript code is executed after the page finished loading but before the page is rendered. This allow to use the JavaScript code to modify the page being rendered. For example, you can do it with Scrapy-Splash:
# Render page and modify its title dynamically
yield SplashRequest(
    'http://example.com',
    endpoint='render.html',
    args={'js_source': 'document.title="My Title";'},
)

Splash Scripts

Splash supports Lua scripts through its execute endpoint. This is the preferred way to execute JavaScript as you can preload libraries, choose when to execute the JavaScript, and retrieve the output.
Here’s an example script:
function main(splash)
    assert(splash:go(splash.args.url))
    splash:wait(0.5)
    local title = splash:evaljs("document.title")
    return {title=title}
end
You need to send that script to the execute endpoint, in the lua_source argument.
This will return a JSON object containing the title:
{
    "title": "Some title"
}
Every script requires a main function to act as the entry point. You can return a Lua table which will be rendered as JSON, which is what we have done here. We use the splash:gofunction to tell Splash to visit the URL. The splash:evaljs function lets you execute JavaScript within the page context, however, if you don't need the result you should use splash:runjsinstead.
You can test your Splash scripts in your browser by visiting your Splash instance’s index page (e.g. http://localhost:8050/). It’s also possible to use Splash with IPython notebook as an interactive web-based development environment, see here for more details.
It’s often the case that you need to click a button before the page is displayed. We can do that using splash:mouse_click function:
function main(splash)
    assert(splash:go(splash.args.url))
    local get_dimensions = splash:jsfunc([[
        function () {
            var rect = document.getElementById('button').getClientRects()[0];
            return {"x": rect.left, "y": rect.top}
        }
    ]])
    splash:set_viewport_full()
    splash:wait(0.1)
    local dimensions = get_dimensions()
    splash:mouse_click(dimensions.x, dimensions.y)
    -- Wait split second to allow event to propagate.
    splash:wait(0.1)
    return splash:html()
end
Here we use splash:jsfunc to define a function that will return the element coordinates, then make sure the element is visible with splash:set_viewport_full, and click on the element. Splash then returns the rendered HTML.
You can find more info on running JavaScript with Splash in the docs, and for a more in-depth tutorial, check out the Splash Scripts Tutorial.
We hope this tutorial gave you a nice introduction to Splash, and please let us know if you have any questions or comments!

Rank

seo