Skip to content
This repository was archived by the owner on Nov 13, 2024. It is now read-only.
This repository was archived by the owner on Nov 13, 2024. It is now read-only.

Training speed #5732

Description

@gauravlokha

Using a gtx 1650 , I know not much at all but it's just for learning.
So usually I leave the training overnight. Usually I get around 350ms give or take and get around 2-3 iterations per second.
Now I am not using my pc since it's training but I get readings of 2900ms. But it's taking 2-3 seconds for each iteration. Can someone explain to me if I'm not getting something right?

update - when I started using my pc to post this, the training has gone back to 350ms and getting usual 2-3 iter per second.

== ==
== Current iteration: 0 ==
== ==
==-------------- Model Options ---------------==
== ==
== batch_size: 2 ==
== ==
==---------------- Running On ----------------==
== ==
== Device index: 0 ==
== Name: NVIDIA GeForce GTX 1650 ==
== VRAM: 2.85GB ==
== ==

Starting. Press "Enter" to stop training and save model.

Trying to do the first iteration. If an error occurs, reduce the model parameters.

!!!
Windows 10 users IMPORTANT notice. You should set this setting in order to work correctly.
https://i.imgur.com/B7cmDCB.jpg
!!!
[02:13:28][#2][0394ms][3.1017][3.3906]
[02:38:14][#3961][0365ms][0.6420][0.5605]
[03:03:14][#007982][0387ms][0.3743][0.3337]
[03:28:14][#011995][0378ms][0.3142][0.2822]
[03:53:14][#016011][0398ms][0.2847][0.2543]
[04:18:14][#020029][0374ms][0.2622][0.2386]
[04:43:14][#024026][0388ms][0.2474][0.2247]
[05:08:14][#028041][0405ms][0.2342][0.2157]
[05:33:14][#032056][0418ms][0.2255][0.2064]
[05:58:14][#036077][0403ms][0.2181][0.2011]
[06:23:14][#040095][0408ms][0.2109][0.1937]
[06:48:14][#044110][0383ms][0.2048][0.1901]
[07:13:14][#048125][0390ms][0.2010][0.1845]
[07:38:14][#052142][0405ms][0.1951][0.1816]
[08:03:14][#056151][0381ms][0.1910][0.1780]
[08:28:14][#060162][0384ms][0.1881][0.1751]
[08:53:14][#064181][0403ms][0.1851][0.1714]
[09:18:14][#068195][0366ms][0.1813][0.1682]
[09:43:14][#072208][0388ms][0.1792][0.1676]
[10:08:14][#076231][0386ms][0.1747][0.1653]
[10:33:14][#080247][0400ms][0.1739][0.1626]
[11:20:13][#082719][2986ms][0.1713][0.1590]
[11:23:17][#082798][2602ms][0.1662][0.1685]
[11:48:16][#083462][1489ms][0.1707][0.1607]
[12:13:00][#085904][0360ms][0.1333][0.1455]

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions